Jul 14 18:08:21.752443 ip-10-0-131-214 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 18:08:21.752456 ip-10-0-131-214 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 18:08:21.752465 ip-10-0-131-214 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 18:08:21.752773 ip-10-0-131-214 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 18:08:31.769496 ip-10-0-131-214 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 18:08:31.769514 ip-10-0-131-214 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot cc506c09c2db48d08484e00eec4d33ad -- Jul 14 18:10:33.438051 ip-10-0-131-214 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.877718 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883840 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883854 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883858 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883862 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883865 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883868 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883871 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:34.904239 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883874 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883877 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883880 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883883 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883886 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883888 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883891 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883894 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883896 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883899 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883903 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883905 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883908 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883911 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883920 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883923 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883926 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883928 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883931 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:34.905159 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883934 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883936 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883939 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883942 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883945 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883947 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883951 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883955 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883959 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883963 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883966 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883971 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883974 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883977 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883980 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883983 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883985 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883988 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883991 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883993 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:34.906354 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883996 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.883998 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884001 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884003 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884014 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884017 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884020 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884022 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884025 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884028 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884031 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884035 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884037 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884039 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884042 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884044 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884047 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884049 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884052 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884055 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:34.907516 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884058 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884060 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884063 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884066 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884069 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884071 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884074 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884076 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884079 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884081 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884084 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884086 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884089 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884091 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884094 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884096 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884099 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884101 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884103 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884106 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:34.908667 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884109 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884114 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884118 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884121 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884124 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884126 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884129 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884131 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884134 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884137 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884139 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884141 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.884144 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886362 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886370 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886374 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886379 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886383 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886387 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:34.909466 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886391 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886394 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886397 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886400 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886403 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886406 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886409 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886412 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886415 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886417 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886420 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886423 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886426 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886429 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886432 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886435 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886437 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886439 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886442 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886445 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:34.910243 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886447 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886450 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886453 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886455 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886458 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886462 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886464 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886467 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886470 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886472 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886475 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886477 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886480 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886482 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886485 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886487 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886490 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886493 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886496 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886498 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:34.911044 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886501 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886504 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886507 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886509 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886512 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886515 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886518 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886520 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886523 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886526 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886529 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886535 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886537 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886540 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886542 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886545 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886547 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886550 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886552 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886555 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:34.911770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886557 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886560 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886562 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886564 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886567 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886570 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886572 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886575 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886578 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886580 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886583 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886585 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886588 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886592 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886594 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886597 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886599 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886602 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886605 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:34.913790 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886607 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886610 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886612 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886615 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886619 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886621 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886624 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886626 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886629 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886631 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886633 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886636 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886638 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.886642 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887025 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887031 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887035 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887038 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887040 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:34.914569 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887043 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887047 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887050 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887053 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887057 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887059 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887062 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887066 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887069 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887072 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887075 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887078 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887081 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887084 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887087 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887091 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887093 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887099 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887101 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:34.917015 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887104 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887106 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887110 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887113 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887115 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887117 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887120 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887122 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887125 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887128 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887131 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887133 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887136 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887139 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887143 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887146 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887149 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887151 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887154 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887156 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:34.917741 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887159 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887161 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887163 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887166 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887168 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887171 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887173 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887176 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887178 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887181 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887185 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887188 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887190 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887193 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887198 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887201 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887204 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887206 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887209 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887212 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:34.919653 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887215 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887217 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887220 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887223 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887225 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887228 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887230 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887233 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887235 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887238 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887241 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887243 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887246 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887248 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887251 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887253 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887256 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887258 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887261 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887264 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:34.920418 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887266 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887268 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887272 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887275 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887277 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887280 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887284 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887286 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887288 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887291 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887293 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887296 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887298 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887301 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.887304 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887379 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887386 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887393 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887397 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887401 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887405 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:10:34.921197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887408 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887412 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887415 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887418 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887422 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887425 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887428 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887431 2567 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887434 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887437 2567 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887440 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887442 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887447 2567 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887452 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887455 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887459 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887462 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887467 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887470 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887473 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887476 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887479 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887482 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887485 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:10:34.922351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887489 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887492 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887496 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887499 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887501 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887504 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887507 2567 flags.go:64] FLAG: --enable-server="true" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887510 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887514 2567 flags.go:64] FLAG: --event-burst="100" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887517 2567 flags.go:64] FLAG: --event-qps="50" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887520 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887523 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887527 2567 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887530 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887533 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887536 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887539 2567 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887542 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887545 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887547 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887550 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887554 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887557 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887560 2567 flags.go:64] FLAG: --feature-gates="" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887564 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:10:34.923111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887568 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887571 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887575 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887577 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887580 2567 flags.go:64] FLAG: --help="false" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887583 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-131-214.ec2.internal" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887586 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887589 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887592 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887595 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887599 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887601 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887604 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887607 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887609 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887612 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887615 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887618 2567 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887621 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887624 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887626 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887629 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887632 2567 flags.go:64] FLAG: --lock-file="" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887635 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:10:34.925379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887637 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887641 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887646 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887648 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887652 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887655 2567 flags.go:64] FLAG: --logging-format="text" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887658 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887661 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887666 2567 flags.go:64] FLAG: --manifest-url="" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887668 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887672 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887675 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887679 2567 flags.go:64] FLAG: --max-pods="110" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887682 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887685 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887688 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887691 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887694 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887697 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887700 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887706 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887709 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887712 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:10:34.926134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887715 2567 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887718 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887724 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887727 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887730 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887733 2567 flags.go:64] FLAG: --port="10250" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887736 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887738 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08ed17f95da074c20" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887741 2567 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887744 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887747 2567 flags.go:64] FLAG: --register-node="true" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887750 2567 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887753 2567 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887756 2567 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887761 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887763 2567 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887767 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887772 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887775 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887778 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887781 2567 flags.go:64] FLAG: --runonce="false" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887784 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887787 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887789 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887792 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:10:34.926904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887795 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887798 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887801 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887804 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887821 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887825 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887828 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887831 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887834 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887838 2567 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887841 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887847 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887850 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887852 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887857 2567 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887859 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887862 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887865 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887868 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887870 2567 flags.go:64] FLAG: --v="2" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887874 2567 flags.go:64] FLAG: --version="false" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887879 2567 flags.go:64] FLAG: --vmodule="" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887883 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.887886 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888022 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:34.927645 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888027 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888030 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888033 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888036 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888039 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888042 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888046 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888050 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888053 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888056 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888058 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888061 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888063 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888066 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888068 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888071 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888074 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888077 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888081 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888084 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:34.929175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888086 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888090 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888092 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888095 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888098 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888101 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888104 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888107 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888112 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888115 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888117 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888121 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888124 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888127 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888130 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888132 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888135 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888137 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888140 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:34.929945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888142 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888145 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888148 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888150 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888153 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888155 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888158 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888160 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888163 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888166 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888168 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888171 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888173 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888176 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888179 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888182 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888184 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888187 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888189 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888191 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:34.931465 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888194 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888198 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888200 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888203 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888206 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888209 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888211 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888214 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888217 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888219 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888222 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888224 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888227 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888229 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888232 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888235 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888237 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888240 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888242 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:34.932213 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888245 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888247 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888250 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888252 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888255 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888258 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888260 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888262 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888265 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888268 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888270 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888272 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888275 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888277 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888282 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888284 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888287 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888291 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888293 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:34.933784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.888296 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.888301 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.896515 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.896534 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.897328 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.897337 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897413 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897419 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897424 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897428 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897431 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897435 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897438 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897441 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897445 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:34.934572 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897448 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897451 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897454 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897458 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897461 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897465 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897468 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897471 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897474 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897477 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897481 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897484 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897487 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897492 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897497 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897501 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897505 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897509 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897513 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:34.936182 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897518 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897521 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897525 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897528 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897531 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897535 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897539 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897544 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897547 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897550 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897554 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897557 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897560 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897564 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897567 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897570 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897574 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897579 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897584 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897588 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:35.028849 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897591 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897595 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897598 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897602 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897605 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897608 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897611 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897614 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897617 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897621 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897624 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897627 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897630 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897634 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897637 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897640 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897644 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897647 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897651 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897654 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:35.030294 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897658 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897662 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897667 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897671 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897674 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897677 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897681 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897684 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897687 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897690 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897694 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897698 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897701 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897704 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897708 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897711 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897714 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897735 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897740 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897743 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:35.031069 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897747 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897751 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897754 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897757 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897761 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897764 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897768 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897771 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897774 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897777 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897781 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.897788 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897947 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897954 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897957 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:10:35.033379 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897961 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897965 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897968 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897972 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897976 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897979 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897982 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897986 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897989 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897993 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897996 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.897999 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898002 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898005 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898008 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898012 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898015 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898018 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898021 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:10:35.034142 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898024 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898027 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898031 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898034 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898038 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898041 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898044 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898048 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898051 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898054 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898058 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898061 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898065 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898070 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898074 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898079 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898083 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898087 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898090 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:10:35.034902 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898094 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898097 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898101 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898104 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898108 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898111 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898115 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898118 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898121 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898124 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898129 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898132 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898135 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898138 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898142 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898145 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898148 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898151 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898154 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:10:35.035643 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898157 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898160 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898164 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898167 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898171 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898174 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898177 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898182 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898185 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898188 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898191 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898195 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898200 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898203 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898206 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898210 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898213 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898216 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898220 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:10:35.037175 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898225 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898229 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898232 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898236 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898241 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898244 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898248 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898251 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898255 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898258 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898261 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898264 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898267 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898271 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898274 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898277 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898280 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898283 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898286 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:10:35.037937 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:34.898289 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.898297 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.898441 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.901746 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.903217 2567 server.go:1019] "Starting client certificate rotation" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.903312 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.903352 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.905980 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.907726 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.921387 2567 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.935499 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:10:35.039448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.969271 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22gwc" Jul 14 18:10:35.039834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:34.987363 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22gwc" Jul 14 18:10:35.748404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.748385 2567 log.go:25] "Validated CRI v1 image API" Jul 14 18:10:35.750045 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.750027 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:10:35.753295 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.753269 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2fd3201-a037-493c-a9ec-fcc3fcb81c0f:/dev/nvme0n1p4 c92df3dc-0205-4d43-858c-46e0fe2e48a9:/dev/nvme0n1p3] Jul 14 18:10:35.753374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.753291 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:10:35.759669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.759530 2567 manager.go:216] Machine: {Timestamp:2026-07-14 18:10:35.758312829 +0000 UTC m=+1.144211273 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098890 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc6439fc93f76e458b1688155f18c SystemUUID:ec2fc643-9fc9-3f76-e458-b1688155f18c BootID:cc506c09-c2db-48d0-8484-e00eec4d33ad Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:78:2f:02:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:78:2f:02:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:53:c6:68:66:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:10:35.759669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.759648 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:10:35.759878 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.759783 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:10:35.762952 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.762913 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:10:35.763098 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.762954 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-214.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:10:35.764314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.764303 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:10:35.764348 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.764317 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:10:35.764376 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.764347 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:10:35.764988 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.764976 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:10:35.766277 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.766265 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:10:35.769345 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769332 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:10:35.769466 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769455 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:10:35.769507 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769488 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:10:35.769507 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769498 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:10:35.769561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769518 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:10:35.769561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769527 2567 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:10:35.769561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769537 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:10:35.769945 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769934 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:10:35.769983 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.769952 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:10:35.772057 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.772041 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:10:35.773472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.773458 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:10:35.773513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.773488 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:10:35.774198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774187 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:10:35.774229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774205 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:10:35.774229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774211 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:10:35.774229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774217 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:10:35.774229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774226 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774232 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774240 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774245 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774251 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774256 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774265 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:10:35.774338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.774271 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:10:35.775832 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.775821 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:10:35.775873 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.775833 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:10:35.779247 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.779225 2567 server.go:1268] "Started kubelet" Jul 14 18:10:35.779775 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.779744 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:10:35.780052 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.780018 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:10:35.780126 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.780089 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:10:35.780911 ip-10-0-131-214 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:10:35.781627 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.781596 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:10:35.781991 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.781873 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:10:35.783916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.783898 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:35.786088 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.786056 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:10:35.786342 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.786327 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:10:35.786342 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.786332 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:10:35.786472 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.786435 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:35.786916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.786897 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:10:35.786916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.786898 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:10:35.787049 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.786925 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:10:35.787049 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.787014 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:10:35.787049 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.787021 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:10:35.789310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.789048 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:10:35.789310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.789066 2567 factory.go:55] Registering systemd factory Jul 14 18:10:35.789310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.789076 2567 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:10:35.789691 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.789571 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:35.790893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.790545 2567 factory.go:153] Registering CRI-O factory Jul 14 18:10:35.790893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.790566 2567 factory.go:223] Registration of the crio container factory successfully Jul 14 18:10:35.790893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.790590 2567 factory.go:103] Registering Raw factory Jul 14 18:10:35.790893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.790611 2567 manager.go:1196] Started watching for new ooms in manager Jul 14 18:10:35.791288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.791268 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:35.791447 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.791432 2567 manager.go:349] Starting recovery of all containers Jul 14 18:10:35.793594 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.793555 2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-214.ec2.internal" not found Jul 14 18:10:35.793732 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.793708 2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-214.ec2.internal\" not found" node="ip-10-0-131-214.ec2.internal" Jul 14 18:10:35.802308 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.802291 2567 manager.go:354] Recovery completed Jul 14 18:10:35.806245 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.806234 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:35.808661 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.808647 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:35.808719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.808706 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:35.808719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.808718 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:35.809554 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.809541 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:10:35.809554 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.809553 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:10:35.809650 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.809569 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:10:35.809679 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.809547 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:35.810611 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.810589 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:35.810698 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.810622 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:35.810698 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.810635 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:35.811003 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.810986 2567 policy_none.go:49] "None policy: Start" Jul 14 18:10:35.811097 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.811009 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:10:35.811097 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.811024 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:10:35.812642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.812625 2567 policy_none.go:47] "Start" Jul 14 18:10:35.817922 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.817782 2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-214.ec2.internal" not found Jul 14 18:10:35.847364 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847348 2567 manager.go:337] "Starting Device Plugin manager" Jul 14 18:10:35.847452 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.847401 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:10:35.847452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847415 2567 server.go:87] "Starting device plugin registration server" Jul 14 18:10:35.847671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847658 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:10:35.847715 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847672 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:10:35.847787 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847771 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:10:35.847924 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847909 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:10:35.847999 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.847934 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:10:35.848381 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.848354 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:10:35.848467 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.848400 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:35.881955 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.881934 2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-214.ec2.internal" not found Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.902215 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.902357 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.902401 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.902403 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.922917 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.924144 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.924187 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.924216 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:10:35.932193 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.924259 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:10:35.935219 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.935200 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:35.948519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.948501 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:35.949477 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.949456 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:35.949573 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.949491 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:35.949573 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.949505 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:35.949573 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.949528 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-214.ec2.internal" Jul 14 18:10:35.964231 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.964167 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-214.ec2.internal" Jul 14 18:10:35.964231 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:35.964201 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-214.ec2.internal\": node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:35.992966 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.992920 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:05:34 +0000 UTC" deadline="2027-12-23 10:15:15.982972941 +0000 UTC" Jul 14 18:10:35.992966 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:35.992961 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12640h4m39.990013826s" Jul 14 18:10:36.016477 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.016455 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.024621 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.024599 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal"] Jul 14 18:10:36.024688 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.024664 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.026317 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.026301 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.026385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.026330 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.026385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.026341 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.028074 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028062 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.028174 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028162 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.028219 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028188 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.028908 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028893 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.028968 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028924 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.028968 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028936 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.028968 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028945 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.028968 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028967 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.029099 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.028981 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.030311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.030298 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.030368 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.030320 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.031063 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.031048 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.031133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.031071 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.031133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.031083 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.052532 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.052508 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-214.ec2.internal\" not found" node="ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.056757 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.056742 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-214.ec2.internal\" not found" node="ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.116869 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.116850 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.188493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.188456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.188493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.188489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.188691 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.188513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a8b05c1a94b1956b000d33abc88ab88-config\") pod \"kube-apiserver-proxy-ip-10-0-131-214.ec2.internal\" (UID: \"3a8b05c1a94b1956b000d33abc88ab88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.217616 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.217554 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.289089 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a8b05c1a94b1956b000d33abc88ab88-config\") pod \"kube-apiserver-proxy-ip-10-0-131-214.ec2.internal\" (UID: \"3a8b05c1a94b1956b000d33abc88ab88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.289089 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.289213 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.289213 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a8b05c1a94b1956b000d33abc88ab88-config\") pod \"kube-apiserver-proxy-ip-10-0-131-214.ec2.internal\" (UID: \"3a8b05c1a94b1956b000d33abc88ab88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.289213 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.289213 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.289163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9c2963d17e820765d656c16a7794d5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal\" (UID: \"fc9c2963d17e820765d656c16a7794d5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.318204 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.318177 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.355388 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.355366 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.360505 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.360393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:36.418330 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.418292 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.518846 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.518784 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.619308 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.619270 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.720014 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.719987 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.770200 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.770148 2567 apiserver.go:52] "Watching apiserver" Jul 14 18:10:36.777729 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.777711 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:10:36.778981 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.778957 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-h6ww8","openshift-network-operator/iptables-alerter-tl24m","openshift-dns/node-resolver-kxk2x","openshift-multus/multus-pxm24","openshift-network-diagnostics/network-check-target-5ksdk","openshift-ovn-kubernetes/ovnkube-node-dmgf9","kube-system/global-pull-secret-syncer-jq6fx","kube-system/konnectivity-agent-bb8mv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8","openshift-cluster-node-tuning-operator/tuned-5cslq","openshift-image-registry/node-ca-vw8tj","openshift-multus/multus-additional-cni-plugins-66tqb"] Jul 14 18:10:36.779066 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.779056 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.779996 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.779975 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.780086 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.780007 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.780086 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.780017 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.781470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.781458 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.781555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.781542 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.781591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.781572 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.782177 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782160 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.782283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782185 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.782283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782196 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.782283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782219 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.782283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782237 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.782446 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.782288 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.783802 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.783788 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.783889 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.783876 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.783939 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.783905 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.784569 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.784539 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.784628 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.784600 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.784692 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.784666 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89hm8\"" Jul 14 18:10:36.785916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785898 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.786016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785927 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.786016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785942 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.786016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785954 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.786016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785972 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.786016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.785984 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.787301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787285 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.787372 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787338 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.787372 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787366 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.787707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787688 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:10:36.787838 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787743 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wgqnn\"" Jul 14 18:10:36.788010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.787991 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:10:36.788130 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788115 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.788179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788144 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.788179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788160 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.788248 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788119 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.788248 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788211 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.788248 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.788214 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:36.788248 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.788222 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.789941 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.789927 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.790013 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790004 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.790072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790033 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.790502 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790487 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.790579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790511 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.790579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790521 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.790579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790558 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.790579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790574 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.790694 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.790584 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.792133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a213abe-36cc-4531-bcb1-012b5312a6a0-konnectivity-ca\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.792133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.792220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg5l\" (UniqueName: \"kubernetes.io/projected/7848077b-6e8d-41bf-a3aa-7e8ec73be770-kube-api-access-9jg5l\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.792220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f330b323-6e28-48a7-b5b9-78d59ac1bf97-hosts-file\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.792220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f330b323-6e28-48a7-b5b9-78d59ac1bf97-tmp-dir\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.792310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a213abe-36cc-4531-bcb1-012b5312a6a0-agent-certs\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.792310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792254 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwct\" (UniqueName: \"kubernetes.io/projected/f330b323-6e28-48a7-b5b9-78d59ac1bf97-kube-api-access-dmwct\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.792310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792266 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.792413 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792326 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.792413 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792351 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.792495 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792424 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2xqc\"" Jul 14 18:10:36.792495 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792436 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.792495 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792474 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.792616 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792540 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:10:36.793012 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.792994 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.793012 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.793008 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.793123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.793026 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.793123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.793027 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.793123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.793044 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.793123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.793050 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.794685 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.794673 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z58pp\"" Jul 14 18:10:36.794774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.794760 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.794822 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.794777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:36.794865 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.794853 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.794897 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.794792 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:10:36.795017 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795002 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.795100 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795086 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.795138 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795098 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:10:36.795418 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795401 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.795489 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795429 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.795489 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795441 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.795552 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.795489 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:36.795552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795440 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.795552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795519 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.795552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.795527 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.797192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797178 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.797265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.797265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797245 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.797920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797902 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.798015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797931 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.798015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797941 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.798015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797902 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.798015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.797997 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.798015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.798008 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.798436 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.798413 2567 predicate.go:218] "Predicate failed on Pod" pod="kube-system/global-pull-secret-syncer-jq6fx" err="Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector" Jul 14 18:10:36.798502 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.798435 2567 allocation_manager.go:702] "Pod admission denied" podUID="15d0673a-1b2e-4365-8022-671e537ca38d" pod="kube-system/global-pull-secret-syncer-jq6fx" reason="NodeAffinity" message="Predicate NodeAffinity failed: node(s) didn't match Pod's node affinity/selector" Jul 14 18:10:36.798669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.798656 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.799558 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.799538 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.799642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.799568 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.799642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.799590 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.800439 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.800420 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:10:36.800577 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.800555 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-57hlz\"" Jul 14 18:10:36.801333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.800760 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.801333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.800760 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:10:36.801333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801124 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:10:36.801333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801130 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.801333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801212 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:10:36.801600 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801581 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.801651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801611 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.801651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.801625 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.802707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802649 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.802707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802649 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.802707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802673 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.802707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802685 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.802948 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802685 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.802948 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.802791 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.804517 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804501 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.804578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.804578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804550 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.804578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804549 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.804801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804779 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:10:36.804894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804881 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.804996 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.804978 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5gnt5\"" Jul 14 18:10:36.805206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805192 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.805283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805211 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.805283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805213 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.805283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805222 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.805283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805233 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.805283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.805243 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.806876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.806864 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:36.806993 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.806885 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.806993 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.806896 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.807199 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807184 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.807273 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807190 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.807352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807338 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2k55\"" Jul 14 18:10:36.807568 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807556 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.807625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807580 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.807625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807559 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.807625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807618 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.807761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807590 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.807761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.807635 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.809078 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.809154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809093 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.809422 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809407 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hqzfk\"" Jul 14 18:10:36.809568 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809543 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:10:36.809651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809636 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:10:36.809727 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809714 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.809780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809718 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:10:36.809780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809740 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.809780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.809762 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.811504 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.811489 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:10:36.811583 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.811533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:10:36.811758 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.811745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4trv\"" Jul 14 18:10:36.820128 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.820114 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.872354 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.872303 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jq6fx" podStartSLOduration=0.87228786 podStartE2EDuration="872.28786ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:36.872176025 +0000 UTC m=+2.258074479" watchObservedRunningTime="2026-07-14 18:10:36.87228786 +0000 UTC m=+2.258186313" Jul 14 18:10:36.888030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.888012 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:10:36.892510 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-kubelet\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.892612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-systemd-units\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.892612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/088fa697-00ba-445f-9501-a613e61118e6-host-slash\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.892612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqqn\" (UniqueName: \"kubernetes.io/projected/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kube-api-access-wkqqn\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.892612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892596 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-modprobe-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-lib-modules\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-var-lib-kubelet\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892692 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-host\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892758 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg5l\" (UniqueName: \"kubernetes.io/projected/7848077b-6e8d-41bf-a3aa-7e8ec73be770-kube-api-access-9jg5l\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-bin\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-tmp\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-system-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892858 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.892891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cni-binary-copy\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-kubelet\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-etc-kubernetes\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cnibin\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.892974 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-binary-copy\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-run\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893030 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-k8s-cni-cncf-io\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-hostroot\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-conf-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smdv\" (UniqueName: \"kubernetes.io/projected/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-kube-api-access-8smdv\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-var-lib-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-system-cni-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-sys-fs\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.893184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a213abe-36cc-4531-bcb1-012b5312a6a0-agent-certs\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893257 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-conf\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893312 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-env-overrides\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893355 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwct\" (UniqueName: \"kubernetes.io/projected/f330b323-6e28-48a7-b5b9-78d59ac1bf97-kube-api-access-dmwct\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a213abe-36cc-4531-bcb1-012b5312a6a0-konnectivity-ca\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-netns\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893435 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f5a69d-b410-48eb-af6a-77592914a97d-serviceca\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893494 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-config\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893533 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-registration-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893574 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-kubernetes\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.893612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lf9\" (UniqueName: \"kubernetes.io/projected/55f5a69d-b410-48eb-af6a-77592914a97d-kube-api-access-m6lf9\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-os-release\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893669 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9f28\" (UniqueName: \"kubernetes.io/projected/df8b3c37-aeb5-4c19-986f-725b81421e2d-kube-api-access-s9f28\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-tuned\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893726 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cnibin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-multus\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893757 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f330b323-6e28-48a7-b5b9-78d59ac1bf97-hosts-file\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-slash\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-node-log\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893881 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893888 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f330b323-6e28-48a7-b5b9-78d59ac1bf97-hosts-file\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893882 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-log-socket\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893908 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893951 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.893973 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-script-lib\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.894344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-sys\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-netd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894147 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894171 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894186 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894198 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894222 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-systemd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894263 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894296 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894299 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8htg\" (UniqueName: \"kubernetes.io/projected/088fa697-00ba-445f-9501-a613e61118e6-kube-api-access-z8htg\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894328 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-socket-dir-parent\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894381 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-os-release\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxp2v\" (UniqueName: \"kubernetes.io/projected/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-kube-api-access-vxp2v\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/088fa697-00ba-445f-9501-a613e61118e6-iptables-alerter-script\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-multus-certs\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894603 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysconfig\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.895083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-socket-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-ovn\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.894678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-device-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.895662 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.895740 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-netns\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.895933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.895856 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f330b323-6e28-48a7-b5b9-78d59ac1bf97-tmp-dir\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.895954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-etc-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.895988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqtw\" (UniqueName: \"kubernetes.io/projected/39ca3d93-2025-4012-9dca-a0fbf802cc52-kube-api-access-zxqtw\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f5a69d-b410-48eb-af6a-77592914a97d-host\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-bin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.896206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.896501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896214 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.896501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-systemd\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.896501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896314 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-daemon-config\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.896629 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.896525 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.897044 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.897020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a213abe-36cc-4531-bcb1-012b5312a6a0-konnectivity-ca\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.897583 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.897566 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:36.897682 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.897593 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:36.897682 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.897608 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:36.898054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.897953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f330b323-6e28-48a7-b5b9-78d59ac1bf97-tmp-dir\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.898054 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.898032 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:36.898215 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.898106 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:10:36.898215 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.898148 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.398119584 +0000 UTC m=+2.784018014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:36.901619 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.901597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a213abe-36cc-4531-bcb1-012b5312a6a0-agent-certs\") pod \"konnectivity-agent-bb8mv\" (UID: \"0a213abe-36cc-4531-bcb1-012b5312a6a0\") " pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:36.904101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.904080 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwct\" (UniqueName: \"kubernetes.io/projected/f330b323-6e28-48a7-b5b9-78d59ac1bf97-kube-api-access-dmwct\") pod \"node-resolver-kxk2x\" (UID: \"f330b323-6e28-48a7-b5b9-78d59ac1bf97\") " pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:36.904191 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.904175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg5l\" (UniqueName: \"kubernetes.io/projected/7848077b-6e8d-41bf-a3aa-7e8ec73be770-kube-api-access-9jg5l\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:36.920186 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:36.920167 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:36.979407 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.979383 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:36.997188 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-config\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997221 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-registration-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-kubernetes\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lf9\" (UniqueName: \"kubernetes.io/projected/55f5a69d-b410-48eb-af6a-77592914a97d-kube-api-access-m6lf9\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-os-release\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-registration-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.997344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997353 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f28\" (UniqueName: \"kubernetes.io/projected/df8b3c37-aeb5-4c19-986f-725b81421e2d-kube-api-access-s9f28\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-tuned\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cnibin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997407 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997412 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-os-release\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-multus\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-kubernetes\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997447 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997437 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997480 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-multus\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997503 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cnibin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-slash\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997515 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997478 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-node-log\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-log-socket\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-slash\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-script-lib\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.997726 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997591 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-node-log\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997610 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-sys\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-log-socket\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997633 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-netd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-systemd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-netd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8htg\" (UniqueName: \"kubernetes.io/projected/088fa697-00ba-445f-9501-a613e61118e6-kube-api-access-z8htg\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-systemd\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997690 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-sys\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997756 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-config\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-socket-dir-parent\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997785 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-socket-dir-parent\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997806 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997850 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.998546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-os-release\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxp2v\" (UniqueName: \"kubernetes.io/projected/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-kube-api-access-vxp2v\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/088fa697-00ba-445f-9501-a613e61118e6-iptables-alerter-script\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997919 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-os-release\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-multus-certs\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-multus-certs\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysconfig\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.997988 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysconfig\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-socket-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998107 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-ovn\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovnkube-script-lib\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-ovn\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-device-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:36.999435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-netns\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998182 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-device-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998168 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-socket-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-etc-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-netns\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqtw\" (UniqueName: \"kubernetes.io/projected/39ca3d93-2025-4012-9dca-a0fbf802cc52-kube-api-access-zxqtw\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998278 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f5a69d-b410-48eb-af6a-77592914a97d-host\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-bin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f5a69d-b410-48eb-af6a-77592914a97d-host\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-etc-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-cni-bin\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998375 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998395 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998439 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-systemd\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.000335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998464 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-daemon-config\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-kubelet\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/088fa697-00ba-445f-9501-a613e61118e6-iptables-alerter-script\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-systemd\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998513 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-systemd-units\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998549 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-systemd-units\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/088fa697-00ba-445f-9501-a613e61118e6-host-slash\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-kubelet\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqqn\" (UniqueName: \"kubernetes.io/projected/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kube-api-access-wkqqn\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998610 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-modprobe-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/088fa697-00ba-445f-9501-a613e61118e6-host-slash\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-lib-modules\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998650 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-var-lib-kubelet\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-host\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998692 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-bin\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-tmp\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.001225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-system-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cni-binary-copy\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-kubelet\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-etc-kubernetes\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998849 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cnibin\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-binary-copy\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-run\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-k8s-cni-cncf-io\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-hostroot\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-daemon-config\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-conf-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smdv\" (UniqueName: \"kubernetes.io/projected/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-kube-api-access-8smdv\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999055 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-var-lib-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999077 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-run\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-system-cni-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-sys-fs\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999181 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.002053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-conf\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-env-overrides\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-netns\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999378 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f5a69d-b410-48eb-af6a-77592914a97d-serviceca\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-etc-kubernetes\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999478 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999505 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.998930 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999520 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999531 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-cni-binary-copy\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-hostroot\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999555 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999570 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999573 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999592 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999605 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-run-k8s-cni-cncf-io\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-system-cni-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999649 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999658 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999674 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999679 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.002928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999691 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999702 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999721 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999767 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999776 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999830 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999845 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-var-lib-kubelet\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-host\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cnibin\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999781 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999889 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999778 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999930 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-lib-modules\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999948 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999954 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-modprobe-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999984 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-sys-fs\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999997 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-var-lib-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-run-openvswitch\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.000107 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999949 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000148 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.000159 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.500142035 +0000 UTC m=+2.886040500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000161 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.003714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000370 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df8b3c37-aeb5-4c19-986f-725b81421e2d-env-overrides\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-cni-bin\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df8b3c37-aeb5-4c19-986f-725b81421e2d-host-run-netns\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000473 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-d\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:36.999856 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000504 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000525 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000538 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000613 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000631 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000644 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-sysctl-conf\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000793 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000830 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000844 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-host-var-lib-kubelet\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000880 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-system-cni-dir\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-multus-conf-dir\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.000996 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-cni-binary-copy\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.001299 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f5a69d-b410-48eb-af6a-77592914a97d-serviceca\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.002975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-etc-tuned\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.004374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.002993 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ca3d93-2025-4012-9dca-a0fbf802cc52-tmp\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.007694 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.007678 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8b3c37-aeb5-4c19-986f-725b81421e2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.014564 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.014542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lf9\" (UniqueName: \"kubernetes.io/projected/55f5a69d-b410-48eb-af6a-77592914a97d-kube-api-access-m6lf9\") pod \"node-ca-vw8tj\" (UID: \"55f5a69d-b410-48eb-af6a-77592914a97d\") " pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.017120 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.017097 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:37.017120 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.017121 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:37.017282 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.017133 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:37.017282 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.017214 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:37.517195644 +0000 UTC m=+2.903094094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:37.019172 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.019147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9f28\" (UniqueName: \"kubernetes.io/projected/df8b3c37-aeb5-4c19-986f-725b81421e2d-kube-api-access-s9f28\") pod \"ovnkube-node-dmgf9\" (UID: \"df8b3c37-aeb5-4c19-986f-725b81421e2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.019249 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.019205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqqn\" (UniqueName: \"kubernetes.io/projected/21d9c12a-db5b-46a7-9a6a-e5e5bce21aee-kube-api-access-wkqqn\") pod \"aws-ebs-csi-driver-node-mqjt8\" (UID: \"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.019546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.019528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8htg\" (UniqueName: \"kubernetes.io/projected/088fa697-00ba-445f-9501-a613e61118e6-kube-api-access-z8htg\") pod \"iptables-alerter-tl24m\" (UID: \"088fa697-00ba-445f-9501-a613e61118e6\") " pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:37.019750 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.019736 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smdv\" (UniqueName: \"kubernetes.io/projected/b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6-kube-api-access-8smdv\") pod \"multus-pxm24\" (UID: \"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6\") " pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.020072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.020054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxp2v\" (UniqueName: \"kubernetes.io/projected/ea73bc14-42d0-4afd-98b5-8163d7ed0a68-kube-api-access-vxp2v\") pod \"multus-additional-cni-plugins-66tqb\" (UID: \"ea73bc14-42d0-4afd-98b5-8163d7ed0a68\") " pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.020134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.020107 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqtw\" (UniqueName: \"kubernetes.io/projected/39ca3d93-2025-4012-9dca-a0fbf802cc52-kube-api-access-zxqtw\") pod \"tuned-5cslq\" (UID: \"39ca3d93-2025-4012-9dca-a0fbf802cc52\") " pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.020959 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.020926 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:37.116259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.116226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxk2x" Jul 14 18:10:37.121955 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.121929 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:37.131136 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.131117 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:10:37.148598 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.148571 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tl24m" Jul 14 18:10:37.167779 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.167751 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxm24" Jul 14 18:10:37.174024 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.174004 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:10:37.202274 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.202238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" Jul 14 18:10:37.213071 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.213047 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cslq" Jul 14 18:10:37.222712 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.222691 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-214.ec2.internal\" not found" Jul 14 18:10:37.229864 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.229836 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ca3d93_2025_4012_9dca_a0fbf802cc52.slice/crio-c5043713e10e5b892ab6b234f7c74148b877b3ef33d61711bfe193176416264e WatchSource:0}: Error finding container c5043713e10e5b892ab6b234f7c74148b877b3ef33d61711bfe193176416264e: Status 404 returned error can't find the container with id c5043713e10e5b892ab6b234f7c74148b877b3ef33d61711bfe193176416264e Jul 14 18:10:37.230397 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.230376 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d9c12a_db5b_46a7_9a6a_e5e5bce21aee.slice/crio-dac30d747005400cebd20e65cc20406f89d533a4175f646073d6f9a58959cd64 WatchSource:0}: Error finding container dac30d747005400cebd20e65cc20406f89d533a4175f646073d6f9a58959cd64: Status 404 returned error can't find the container with id dac30d747005400cebd20e65cc20406f89d533a4175f646073d6f9a58959cd64 Jul 14 18:10:37.230997 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.230977 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8b05c1a94b1956b000d33abc88ab88.slice/crio-dfb79faaa12d7d4bdb34bd122ef822abe6a72478fdff0c1c99cc5f2b4e4da7f5 WatchSource:0}: Error finding container dfb79faaa12d7d4bdb34bd122ef822abe6a72478fdff0c1c99cc5f2b4e4da7f5: Status 404 returned error can't find the container with id dfb79faaa12d7d4bdb34bd122ef822abe6a72478fdff0c1c99cc5f2b4e4da7f5 Jul 14 18:10:37.237391 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.237376 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:10:37.247064 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.247036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vw8tj" Jul 14 18:10:37.251571 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.251550 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66tqb" Jul 14 18:10:37.256861 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.256836 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f5a69d_b410_48eb_af6a_77592914a97d.slice/crio-b44eddff6a0102b5c4923db01e6297e6cdf284a086d3336ba12cf7248c66f713 WatchSource:0}: Error finding container b44eddff6a0102b5c4923db01e6297e6cdf284a086d3336ba12cf7248c66f713: Status 404 returned error can't find the container with id b44eddff6a0102b5c4923db01e6297e6cdf284a086d3336ba12cf7248c66f713 Jul 14 18:10:37.260607 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.260589 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea73bc14_42d0_4afd_98b5_8163d7ed0a68.slice/crio-d6c4846c24afa9de6450ab219b8fe148c45fed08140c1585acef2ef90c8874af WatchSource:0}: Error finding container d6c4846c24afa9de6450ab219b8fe148c45fed08140c1585acef2ef90c8874af: Status 404 returned error can't find the container with id d6c4846c24afa9de6450ab219b8fe148c45fed08140c1585acef2ef90c8874af Jul 14 18:10:37.320973 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.320895 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:37.386610 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.386580 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" Jul 14 18:10:37.402032 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.402006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:37.402196 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.402161 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:37.402234 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.402224 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:38.402207622 +0000 UTC m=+3.788106058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:37.406706 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.406685 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:10:37.406851 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.406768 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:10:37.406915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.406860 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" Jul 14 18:10:37.407359 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.407338 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal"] Jul 14 18:10:37.416336 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.416315 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:10:37.416760 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.416743 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal"] Jul 14 18:10:37.465894 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.465866 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a213abe_36cc_4531_bcb1_012b5312a6a0.slice/crio-419c55f9c14dc7733d395c7070ca70486a7c3ab4ab27a1c186c870034e8dece1 WatchSource:0}: Error finding container 419c55f9c14dc7733d395c7070ca70486a7c3ab4ab27a1c186c870034e8dece1: Status 404 returned error can't find the container with id 419c55f9c14dc7733d395c7070ca70486a7c3ab4ab27a1c186c870034e8dece1 Jul 14 18:10:37.502752 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.502728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:37.502863 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.502850 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:37.502913 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.502897 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:10:38.502884613 +0000 UTC m=+3.888783044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:37.603711 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.603625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:37.603888 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.603782 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:37.603888 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.603803 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:37.603888 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.603829 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:37.604011 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:37.603893 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:38.603874145 +0000 UTC m=+3.989772593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:37.606839 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.606799 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30c9a26_d4e9_4b8d_a1e8_fe0f50c389c6.slice/crio-6cd2d27e1c18e07805bd69ae2bb7d07fb1d4d2881f3077b0df0ee1cf3c90e834 WatchSource:0}: Error finding container 6cd2d27e1c18e07805bd69ae2bb7d07fb1d4d2881f3077b0df0ee1cf3c90e834: Status 404 returned error can't find the container with id 6cd2d27e1c18e07805bd69ae2bb7d07fb1d4d2881f3077b0df0ee1cf3c90e834 Jul 14 18:10:37.652680 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.652645 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8b3c37_aeb5_4c19_986f_725b81421e2d.slice/crio-0270467fdd91c0d5bdd947637d9e68ec85c5740d2ca5d1bd7532d3f49ab4e80d WatchSource:0}: Error finding container 0270467fdd91c0d5bdd947637d9e68ec85c5740d2ca5d1bd7532d3f49ab4e80d: Status 404 returned error can't find the container with id 0270467fdd91c0d5bdd947637d9e68ec85c5740d2ca5d1bd7532d3f49ab4e80d Jul 14 18:10:37.685023 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.684998 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088fa697_00ba_445f_9501_a613e61118e6.slice/crio-40e5da4249261457c6825241077ca40217a0ca77e06ff69afa95a7b45ccb0f42 WatchSource:0}: Error finding container 40e5da4249261457c6825241077ca40217a0ca77e06ff69afa95a7b45ccb0f42: Status 404 returned error can't find the container with id 40e5da4249261457c6825241077ca40217a0ca77e06ff69afa95a7b45ccb0f42 Jul 14 18:10:37.787312 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.787281 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:10:37.800984 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.800940 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:10:37.821171 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:37.821128 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9c2963d17e820765d656c16a7794d5.slice/crio-b0d06b09a2d4e3eabd5c131999f485e32afeeacd65bcec77a7e18160c830b5d0 WatchSource:0}: Error finding container b0d06b09a2d4e3eabd5c131999f485e32afeeacd65bcec77a7e18160c830b5d0: Status 404 returned error can't find the container with id b0d06b09a2d4e3eabd5c131999f485e32afeeacd65bcec77a7e18160c830b5d0 Jul 14 18:10:37.838366 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.838342 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cszzp" Jul 14 18:10:37.848304 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.848274 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cszzp" Jul 14 18:10:37.929515 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.929406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tl24m" event={"ID":"088fa697-00ba-445f-9501-a613e61118e6","Type":"ContainerStarted","Data":"40e5da4249261457c6825241077ca40217a0ca77e06ff69afa95a7b45ccb0f42"} Jul 14 18:10:37.930951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.930923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb8mv" event={"ID":"0a213abe-36cc-4531-bcb1-012b5312a6a0","Type":"ContainerStarted","Data":"419c55f9c14dc7733d395c7070ca70486a7c3ab4ab27a1c186c870034e8dece1"} Jul 14 18:10:37.932335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.932311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" event={"ID":"fc9c2963d17e820765d656c16a7794d5","Type":"ContainerStarted","Data":"b0d06b09a2d4e3eabd5c131999f485e32afeeacd65bcec77a7e18160c830b5d0"} Jul 14 18:10:37.934591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.934566 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"0270467fdd91c0d5bdd947637d9e68ec85c5740d2ca5d1bd7532d3f49ab4e80d"} Jul 14 18:10:37.936092 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.936032 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxm24" event={"ID":"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6","Type":"ContainerStarted","Data":"6cd2d27e1c18e07805bd69ae2bb7d07fb1d4d2881f3077b0df0ee1cf3c90e834"} Jul 14 18:10:37.937735 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.937712 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerStarted","Data":"d6c4846c24afa9de6450ab219b8fe148c45fed08140c1585acef2ef90c8874af"} Jul 14 18:10:37.939180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.939155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vw8tj" event={"ID":"55f5a69d-b410-48eb-af6a-77592914a97d","Type":"ContainerStarted","Data":"b44eddff6a0102b5c4923db01e6297e6cdf284a086d3336ba12cf7248c66f713"} Jul 14 18:10:37.941107 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.941084 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cslq" event={"ID":"39ca3d93-2025-4012-9dca-a0fbf802cc52","Type":"ContainerStarted","Data":"c5043713e10e5b892ab6b234f7c74148b877b3ef33d61711bfe193176416264e"} Jul 14 18:10:37.944363 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.942909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" event={"ID":"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee","Type":"ContainerStarted","Data":"dac30d747005400cebd20e65cc20406f89d533a4175f646073d6f9a58959cd64"} Jul 14 18:10:37.944363 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:37.944253 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" event={"ID":"3a8b05c1a94b1956b000d33abc88ab88","Type":"ContainerStarted","Data":"dfb79faaa12d7d4bdb34bd122ef822abe6a72478fdff0c1c99cc5f2b4e4da7f5"} Jul 14 18:10:38.005966 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.005928 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config\") pod \"15d0673a-1b2e-4365-8022-671e537ca38d\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " Jul 14 18:10:38.006780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.006012 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus\") pod \"15d0673a-1b2e-4365-8022-671e537ca38d\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " Jul 14 18:10:38.006780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.006202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus" (OuterVolumeSpecName: "dbus") pod "15d0673a-1b2e-4365-8022-671e537ca38d" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d"). InnerVolumeSpecName "dbus". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Jul 14 18:10:38.006780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.006246 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config" (OuterVolumeSpecName: "kubelet-config") pod "15d0673a-1b2e-4365-8022-671e537ca38d" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d"). InnerVolumeSpecName "kubelet-config". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Jul 14 18:10:38.108598 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.108566 2567 reconciler_common.go:299] "Volume detached for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-dbus\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:10:38.108598 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.108600 2567 reconciler_common.go:299] "Volume detached for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15d0673a-1b2e-4365-8022-671e537ca38d-kubelet-config\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:10:38.411056 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.411004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:38.411250 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.411184 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:38.411250 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.411245 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.411227569 +0000 UTC m=+5.797126006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:38.512041 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.512004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:38.512222 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.512204 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:38.512285 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.512267 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.512248749 +0000 UTC m=+5.898147182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:38.577188 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:10:38.577155 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf330b323_6e28_48a7_b5b9_78d59ac1bf97.slice/crio-9daaa64ebd7b449316f63ba2c9a09bf969100e47a13b8b6bbeb81e40cef06d80 WatchSource:0}: Error finding container 9daaa64ebd7b449316f63ba2c9a09bf969100e47a13b8b6bbeb81e40cef06d80: Status 404 returned error can't find the container with id 9daaa64ebd7b449316f63ba2c9a09bf969100e47a13b8b6bbeb81e40cef06d80 Jul 14 18:10:38.612752 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.612718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:38.612975 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.612945 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:38.612975 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.612969 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:38.613143 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.612982 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:38.613143 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.613039 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:40.613020647 +0000 UTC m=+5.998919102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:38.851314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.851214 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:05:37 +0000 UTC" deadline="2028-03-31 06:39:32.878570474 +0000 UTC" Jul 14 18:10:38.851314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.851314 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15012h28m54.027262232s" Jul 14 18:10:38.925805 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.924963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:38.925805 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.925115 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:38.925805 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.925148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:38.926119 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:38.925912 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:38.961240 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:38.961204 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxk2x" event={"ID":"f330b323-6e28-48a7-b5b9-78d59ac1bf97","Type":"ContainerStarted","Data":"9daaa64ebd7b449316f63ba2c9a09bf969100e47a13b8b6bbeb81e40cef06d80"} Jul 14 18:10:39.852434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:39.852358 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:05:37 +0000 UTC" deadline="2028-03-01 23:23:50.583905891 +0000 UTC" Jul 14 18:10:39.852434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:39.852397 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14309h13m10.731513155s" Jul 14 18:10:40.429745 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:40.429699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:40.429953 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.429901 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:40.430019 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.429964 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:44.429945058 +0000 UTC m=+9.815843492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:40.531166 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:40.530539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:40.531166 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.530730 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:40.531166 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.530795 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:10:44.530777735 +0000 UTC m=+9.916676167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:40.632049 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:40.632010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:40.632229 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.632184 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:40.632229 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.632203 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:40.632229 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.632215 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:40.632401 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.632278 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:44.632258392 +0000 UTC m=+10.018156837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:40.925281 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:40.924595 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:40.925281 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.924740 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:40.925281 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:40.925128 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:40.925281 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:40.925233 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:42.925336 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:42.925266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:42.925786 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:42.925405 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:42.925786 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:42.925449 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:42.925786 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:42.925533 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:44.466951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:44.466912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:44.467441 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.467112 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:44.467441 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.467197 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:52.467174106 +0000 UTC m=+17.853072545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:44.567383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:44.567346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:44.567577 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.567504 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:44.567577 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.567572 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:10:52.567552526 +0000 UTC m=+17.953450960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:44.668876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:44.668528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:44.668876 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.668763 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:44.668876 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.668782 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:44.668876 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.668795 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:44.668876 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.668868 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:10:52.668849763 +0000 UTC m=+18.054748202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:44.924732 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:44.924683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:44.924969 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.924828 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:44.925357 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:44.924683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:44.925357 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:44.925318 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:46.925180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:46.925145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:46.925677 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:46.925629 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:46.925802 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:46.925694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:46.925891 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:46.925801 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:48.925128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:48.925097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:48.925128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:48.925136 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:48.925561 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:48.925228 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:48.925561 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:48.925342 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:50.924863 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:50.924806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:50.925330 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:50.924821 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:50.925330 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:50.924948 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:50.925330 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:50.925022 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:52.531468 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:52.531256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:52.531879 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.531416 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:52.531879 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.531580 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:08.531562209 +0000 UTC m=+33.917460641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:10:52.631788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:52.631749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") pod \"global-pull-secret-syncer-jq6fx\" (UID: \"15d0673a-1b2e-4365-8022-671e537ca38d\") " pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:10:52.631950 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.631873 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:52.631950 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.631933 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret podName:15d0673a-1b2e-4365-8022-671e537ca38d nodeName:}" failed. No retries permitted until 2026-07-14 18:11:08.631914313 +0000 UTC m=+34.017812747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret") pod "global-pull-secret-syncer-jq6fx" (UID: "15d0673a-1b2e-4365-8022-671e537ca38d") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:10:52.732894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:52.732862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:52.733049 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.733031 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:10:52.733091 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.733056 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:10:52.733091 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.733066 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:52.733206 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.733111 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:08.73309855 +0000 UTC m=+34.118996984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:10:52.924876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:52.924837 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:52.924876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:52.924869 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:52.925120 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.924958 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:52.925120 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:52.925062 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:54.925252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:54.925211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:54.925252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:54.925235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:54.925757 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:54.925361 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:54.925757 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:54.925478 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:56.925081 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:56.925048 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:56.925486 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:56.925052 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:56.925486 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:56.925168 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:56.925486 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:56.925275 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:58.016521 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.016281 2567 generic.go:356] "Generic (PLEG): container finished" podID="fc9c2963d17e820765d656c16a7794d5" containerID="21e2b5e9184068d410ca006112ba4771daa1496228af81b856609f633582e4a3" exitCode=0 Jul 14 18:10:58.017428 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.016364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" event={"ID":"fc9c2963d17e820765d656c16a7794d5","Type":"ContainerDied","Data":"21e2b5e9184068d410ca006112ba4771daa1496228af81b856609f633582e4a3"} Jul 14 18:10:58.018739 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.018719 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:10:58.019077 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.019057 2567 generic.go:356] "Generic (PLEG): container finished" podID="df8b3c37-aeb5-4c19-986f-725b81421e2d" containerID="1ebe7053044a4ace26c4f032fbcf5000cd40ac9607ba77c47184f80612192b55" exitCode=1 Jul 14 18:10:58.019141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.019119 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"df98db6348c0c8cdb465caac175849f4b79c1021fce6e1364a2d4ed3a1b850ff"} Jul 14 18:10:58.019195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.019175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"b7ea6d9b25c19c172e33796715f3f8e5c17e825d3f27d4f31e512bc837da97b5"} Jul 14 18:10:58.019235 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.019209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerDied","Data":"1ebe7053044a4ace26c4f032fbcf5000cd40ac9607ba77c47184f80612192b55"} Jul 14 18:10:58.019272 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.019246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"2de2a038a9904652fff8653ab47627af7ae9862c609a91ee06bf1338071031bb"} Jul 14 18:10:58.020582 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.020561 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxm24" event={"ID":"b30c9a26-d4e9-4b8d-a1e8-fe0f50c389c6","Type":"ContainerStarted","Data":"e0988e082aacd97f6b9b24d3dfb1e3d547d83dc742b03f33ddb6b83de0134b84"} Jul 14 18:10:58.022192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.022174 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="7f30cb571b294ce91d3d1c852b955919f70fb62ad780b6aefa04be7c9bee30d0" exitCode=0 Jul 14 18:10:58.022280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.022232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"7f30cb571b294ce91d3d1c852b955919f70fb62ad780b6aefa04be7c9bee30d0"} Jul 14 18:10:58.023866 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.023841 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vw8tj" event={"ID":"55f5a69d-b410-48eb-af6a-77592914a97d","Type":"ContainerStarted","Data":"4fc6b0dc12aa64396b87a9d632767d403df9786a93f50a047eefb876b0fd49fc"} Jul 14 18:10:58.025546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.025521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cslq" event={"ID":"39ca3d93-2025-4012-9dca-a0fbf802cc52","Type":"ContainerStarted","Data":"7b24a549575d41a4a9a3131f8db9c2772e4a7722ba33fbcb8fd8c24610e57740"} Jul 14 18:10:58.027081 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.027033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" event={"ID":"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee","Type":"ContainerStarted","Data":"1f0ddbaf5940caa4f96d83289ca09506a65bdd660934becc7213f9b5f466527d"} Jul 14 18:10:58.028447 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.028417 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" event={"ID":"3a8b05c1a94b1956b000d33abc88ab88","Type":"ContainerStarted","Data":"7b50a6f7f03c3ef693fc66d0b01190337fcc103d809d159b8968e352a2efa498"} Jul 14 18:10:58.029754 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.029735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb8mv" event={"ID":"0a213abe-36cc-4531-bcb1-012b5312a6a0","Type":"ContainerStarted","Data":"053aa3b612daa781b1e707c999f6ca03fd8588a6d70ffc3c3e15e4d3210fc5c9"} Jul 14 18:10:58.031053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.031037 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxk2x" event={"ID":"f330b323-6e28-48a7-b5b9-78d59ac1bf97","Type":"ContainerStarted","Data":"c1f277ff79156d1c676b89d2e300de5180c4aed9f43a0c06f1192752a4787297"} Jul 14 18:10:58.053393 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.053343 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vw8tj" podStartSLOduration=6.669530194 podStartE2EDuration="22.053328892s" totalImagesPullingTime="15.383798698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.053249136 +0000 UTC m=+23.439147580" watchObservedRunningTime="2026-07-14 18:10:58.053328892 +0000 UTC m=+23.439227339" Jul 14 18:10:58.098865 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.098656 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5cslq" podStartSLOduration=2.155724653 podStartE2EDuration="22.09864098s" totalImagesPullingTime="19.942916327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.098310477 +0000 UTC m=+23.484208931" watchObservedRunningTime="2026-07-14 18:10:58.09864098 +0000 UTC m=+23.484539441" Jul 14 18:10:58.117806 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.117762 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pxm24" podStartSLOduration=2.100553185 podStartE2EDuration="22.11774777s" totalImagesPullingTime="20.017194585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.117743124 +0000 UTC m=+23.503641575" watchObservedRunningTime="2026-07-14 18:10:58.11774777 +0000 UTC m=+23.503646220" Jul 14 18:10:58.134066 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.134023 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-214.ec2.internal" podStartSLOduration=21.134007184 podStartE2EDuration="21.134007184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.133640458 +0000 UTC m=+23.519538912" watchObservedRunningTime="2026-07-14 18:10:58.134007184 +0000 UTC m=+23.519905628" Jul 14 18:10:58.180291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.180233 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bb8mv" podStartSLOduration=3.518974536 podStartE2EDuration="23.18021169s" totalImagesPullingTime="19.661237154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.15462034 +0000 UTC m=+23.540518794" watchObservedRunningTime="2026-07-14 18:10:58.18021169 +0000 UTC m=+23.566110142" Jul 14 18:10:58.186492 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.186463 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:10:58.924870 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.924832 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:10:58.924870 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:58.924861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:10:58.925121 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:58.924968 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:10:58.925121 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:10:58.925075 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:10:59.037303 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.037268 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" event={"ID":"fc9c2963d17e820765d656c16a7794d5","Type":"ContainerStarted","Data":"a7880e7fc3b6f3cd5c7474887986d3605aedee7a5ad91330cfc232ede13dcd21"} Jul 14 18:10:59.039965 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.039945 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:10:59.040317 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.040293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"8a96b5357d9b5d0905d3f9b314e9f20eab58b3d9685bdedf59d550157c5a904f"} Jul 14 18:10:59.040386 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.040355 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"b8f41deb57a3816014ce0ecabe1e0540fe2fae02d120958a80b3734ed4acef91"} Jul 14 18:10:59.041600 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.041573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tl24m" event={"ID":"088fa697-00ba-445f-9501-a613e61118e6","Type":"ContainerStarted","Data":"0caf12540bfe119973a3059bd492f03ae2ee20dc0b773c0a7148ad7b7b826be8"} Jul 14 18:10:59.063610 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.063563 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kxk2x" podStartSLOduration=5.467531115 podStartE2EDuration="24.06354783s" totalImagesPullingTime="18.596016715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:58.179795599 +0000 UTC m=+23.565694177" watchObservedRunningTime="2026-07-14 18:10:59.06354783 +0000 UTC m=+24.449446284" Jul 14 18:10:59.063872 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.063850 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-214.ec2.internal" podStartSLOduration=22.063843501 podStartE2EDuration="22.063843501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:59.06366884 +0000 UTC m=+24.449567275" watchObservedRunningTime="2026-07-14 18:10:59.063843501 +0000 UTC m=+24.449741953" Jul 14 18:10:59.923611 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:10:59.923576 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:11:00.045391 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.045299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" event={"ID":"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee","Type":"ContainerStarted","Data":"9ab95c448301c65a80765497f200e358a8345a08db4b2fd8df63018329f87401"} Jul 14 18:11:00.859954 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.859802 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:10:59.923598531Z","UUID":"5b39544d-b41e-46ec-a7a8-f2a6852df89c","Handler":null,"Name":"","Endpoint":""} Jul 14 18:11:00.861406 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.861384 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:11:00.861520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.861429 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:11:00.925151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.925120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:00.925311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:00.925129 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:00.925311 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:00.925267 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:00.925426 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:00.925342 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:01.002907 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.002870 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:11:01.003478 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.003453 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:11:01.022035 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.021960 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tl24m" podStartSLOduration=5.529409208 podStartE2EDuration="25.02194051s" totalImagesPullingTime="19.492531302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:59.081460655 +0000 UTC m=+24.467359355" watchObservedRunningTime="2026-07-14 18:11:01.02194051 +0000 UTC m=+26.407838962" Jul 14 18:11:01.051870 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.051836 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:11:01.052368 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.052337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"36450df43e7588a92c06de7d283938f9e379cb832ee6a71c9626a077fb683870"} Jul 14 18:11:01.052501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.052441 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:11:01.053309 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:01.053289 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bb8mv" Jul 14 18:11:02.056373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:02.056340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" event={"ID":"21d9c12a-db5b-46a7-9a6a-e5e5bce21aee","Type":"ContainerStarted","Data":"10c0c979986bcdc9479d9f7beed1ccb125fa4b8d9849984dbaa75351d645a8f5"} Jul 14 18:11:02.080185 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:02.080136 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqjt8" podStartSLOduration=2.936666774 podStartE2EDuration="26.080117708s" totalImagesPullingTime="23.143450934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:02.079925306 +0000 UTC m=+27.465823777" watchObservedRunningTime="2026-07-14 18:11:02.080117708 +0000 UTC m=+27.466016161" Jul 14 18:11:02.925853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:02.925468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:02.925853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:02.925529 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:02.926206 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:02.925861 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:02.926206 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:02.925992 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:03.063518 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.063492 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:11:03.064324 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.064033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"7ffea26a79c7bb5c31cc899fcaa984a35af63f4496b64964d906bc568b4f5fdd"} Jul 14 18:11:03.064475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.064448 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:03.064557 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.064520 2567 scope.go:117] "RemoveContainer" containerID="1ebe7053044a4ace26c4f032fbcf5000cd40ac9607ba77c47184f80612192b55" Jul 14 18:11:03.064691 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.064646 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:03.064749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.064721 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:03.081485 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.081028 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:03.081485 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.081257 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:03.154276 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.154248 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kube-system/global-pull-secret-syncer-jq6fx"] Jul 14 18:11:03.154438 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.154337 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:11:03.164588 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.164554 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kube-system/global-pull-secret-syncer-jq6fx"] Jul 14 18:11:03.169793 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.169751 2567 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"15d0673a-1b2e-4365-8022-671e537ca38d\\\"},\\\"status\\\":{\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-14T18:11:03Z\\\",\\\"observedGeneration\\\":2,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-14T18:11:03Z\\\",\\\"observedGeneration\\\":2,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-14T18:11:03Z\\\",\\\"observedGeneration\\\":2,\\\"reason\\\":\\\"PodFailed\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-14T18:11:03Z\\\",\\\"observedGeneration\\\":2,\\\"reason\\\":\\\"PodFailed\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-14T18:11:03Z\\\",\\\"observedGeneration\\\":2,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9e719bb83e4c1c822400d7684dc72732516d3564628cdb86c3940e5db30fb54\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"global-pull-secret-syncer\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/lib/kubelet\\\",\\\"name\\\":\\\"kubelet-config\\\"},{\\\"mountPath\\\":\\\"/var/run/dbus\\\",\\\"name\\\":\\\"dbus\\\"},{\\\"mountPath\\\":\\\"/etc/original-pull-secret\\\",\\\"name\\\":\\\"original-pull-secret\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.131.214\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.131.214\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kube-system\"/\"global-pull-secret-syncer-jq6fx\": pods \"global-pull-secret-syncer-jq6fx\" not found" pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:11:03.171304 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.171281 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"global-pull-secret-syncer-jq6fx\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="15d0673a-1b2e-4365-8022-671e537ca38d" pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:11:03.245909 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.245876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:03.291204 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.291173 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-flxqs"] Jul 14 18:11:03.299545 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.299512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.299673 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.299588 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flxqs" podUID="e0e4e0ae-937a-401d-a47c-6691c6926b23" Jul 14 18:11:03.301342 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.301314 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"global-pull-secret-syncer-jq6fx\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="15d0673a-1b2e-4365-8022-671e537ca38d" pod="kube-system/global-pull-secret-syncer-jq6fx" Jul 14 18:11:03.309871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.309847 2567 reconciler_common.go:299] "Volume detached for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15d0673a-1b2e-4365-8022-671e537ca38d-original-pull-secret\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:03.386971 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.386937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:03.411021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.410990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-dbus\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.411176 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.411029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-kubelet-config\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.411176 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.411099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.479826 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.479783 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:03.512250 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.512218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.512419 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.512276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-dbus\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.512419 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.512302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-kubelet-config\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.512419 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.512371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-kubelet-config\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.512419 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.512386 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:03.512610 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:03.512446 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret podName:e0e4e0ae-937a-401d-a47c-6691c6926b23 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:04.012427223 +0000 UTC m=+29.398325654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret") pod "global-pull-secret-syncer-flxqs" (UID: "e0e4e0ae-937a-401d-a47c-6691c6926b23") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:03.512610 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.512544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0e4e0ae-937a-401d-a47c-6691c6926b23-dbus\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:03.601214 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.601180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:03.704223 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:03.704192 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:04.015541 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.015506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:04.015710 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:04.015643 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:04.015756 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:04.015718 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret podName:e0e4e0ae-937a-401d-a47c-6691c6926b23 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:05.015687002 +0000 UTC m=+30.401585437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret") pod "global-pull-secret-syncer-flxqs" (UID: "e0e4e0ae-937a-401d-a47c-6691c6926b23") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:04.069983 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.069957 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:11:04.070521 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.070333 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" event={"ID":"df8b3c37-aeb5-4c19-986f-725b81421e2d","Type":"ContainerStarted","Data":"a5763e0c3aa1044eef97dc702f6594563cfd3a9dd2aa0df2c4a23fcbeab00706"} Jul 14 18:11:04.075148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.075123 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="cfbd525f13efacfda906f8ba5fbdf785948bf6ada88c05e9847c80145ebd952e" exitCode=0 Jul 14 18:11:04.075292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.075169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"cfbd525f13efacfda906f8ba5fbdf785948bf6ada88c05e9847c80145ebd952e"} Jul 14 18:11:04.110282 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.110167 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" podStartSLOduration=8.28699412 podStartE2EDuration="28.110153246s" totalImagesPullingTime="19.823159126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:04.109844488 +0000 UTC m=+29.495742941" watchObservedRunningTime="2026-07-14 18:11:04.110153246 +0000 UTC m=+29.496051699" Jul 14 18:11:04.683318 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.683281 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:04.924867 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.924666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:04.925069 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.924703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:04.925069 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:04.924963 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:04.925069 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:04.924708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:04.925069 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:04.925045 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:04.925284 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:04.925101 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flxqs" podUID="e0e4e0ae-937a-401d-a47c-6691c6926b23" Jul 14 18:11:05.000537 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.000508 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flxqs"] Jul 14 18:11:05.005747 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.005723 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ksdk"] Jul 14 18:11:05.022122 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.022092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:05.022251 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:05.022236 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:05.022309 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:05.022298 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret podName:e0e4e0ae-937a-401d-a47c-6691c6926b23 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:07.022282048 +0000 UTC m=+32.408180485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret") pod "global-pull-secret-syncer-flxqs" (UID: "e0e4e0ae-937a-401d-a47c-6691c6926b23") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:05.028436 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.028410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6ww8"] Jul 14 18:11:05.073504 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.073474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:05.076906 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.076886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:05.077016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.076997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:05.077016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.077007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:05.077126 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:05.077010 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flxqs" podUID="e0e4e0ae-937a-401d-a47c-6691c6926b23" Jul 14 18:11:05.077126 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:05.077099 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:05.077225 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:05.077173 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:05.173698 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:05.173665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.081175 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.081137 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="aed3731a55a3076bdbab65b6eaf8de2ee7bbd9b543dcb7d2b7c64cf8be65824d" exitCode=0 Jul 14 18:11:06.081554 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.081204 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"aed3731a55a3076bdbab65b6eaf8de2ee7bbd9b543dcb7d2b7c64cf8be65824d"} Jul 14 18:11:06.109954 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.109922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.206765 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.206730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.307866 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.307830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.416915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.416829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.695695 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.695610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:06.924888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.924853 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:06.925061 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.924964 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:06.925134 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:06.924967 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:06.925134 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:06.925103 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:06.925134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:06.925132 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:06.925225 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:06.925212 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flxqs" podUID="e0e4e0ae-937a-401d-a47c-6691c6926b23" Jul 14 18:11:07.038172 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:07.038141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:07.038320 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:07.038247 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:07.038320 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:07.038309 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret podName:e0e4e0ae-937a-401d-a47c-6691c6926b23 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:11.038294473 +0000 UTC m=+36.424192904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret") pod "global-pull-secret-syncer-flxqs" (UID: "e0e4e0ae-937a-401d-a47c-6691c6926b23") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:11:08.088294 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.088261 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="ba8c1f2ddc03c963f27b9942a92c3a22010e7ac15f418798dfcf84954227516a" exitCode=0 Jul 14 18:11:08.088690 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.088324 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"ba8c1f2ddc03c963f27b9942a92c3a22010e7ac15f418798dfcf84954227516a"} Jul 14 18:11:08.549973 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.549931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:08.550179 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.550075 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:11:08.550179 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.550149 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs podName:7848077b-6e8d-41bf-a3aa-7e8ec73be770 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:40.550128889 +0000 UTC m=+65.936027327 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs") pod "network-metrics-daemon-h6ww8" (UID: "7848077b-6e8d-41bf-a3aa-7e8ec73be770") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:11:08.751633 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.751598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:08.751798 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.751769 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:11:08.751798 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.751793 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:11:08.751899 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.751820 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6xpqs for pod openshift-network-diagnostics/network-check-target-5ksdk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:11:08.751899 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.751886 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs podName:4aed1a3f-e696-4a12-ab89-ce7a961c4684 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:40.751866182 +0000 UTC m=+66.137764616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xpqs" (UniqueName: "kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs") pod "network-check-target-5ksdk" (UID: "4aed1a3f-e696-4a12-ab89-ce7a961c4684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:11:08.925961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.925849 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:08.925961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.925871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:08.926288 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.925999 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6ww8" podUID="7848077b-6e8d-41bf-a3aa-7e8ec73be770" Jul 14 18:11:08.926288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:08.926049 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:08.926288 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.926120 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5ksdk" podUID="4aed1a3f-e696-4a12-ab89-ce7a961c4684" Jul 14 18:11:08.926288 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:08.926202 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flxqs" podUID="e0e4e0ae-937a-401d-a47c-6691c6926b23" Jul 14 18:11:09.146874 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:09.146842 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:09.215571 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:09.215478 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:10.420356 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.420275 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-214.ec2.internal" event="NodeReady" Jul 14 18:11:10.420801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.420438 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:11:10.484462 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.484431 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4jp2"] Jul 14 18:11:10.486962 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.486844 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.487472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.487448 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8st4g"] Jul 14 18:11:10.489195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489145 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-266xb\"" Jul 14 18:11:10.489195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489179 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:11:10.489195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489195 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:11:10.489195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489242 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:11:10.489652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489146 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:11:10.489652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.489478 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.491764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.491667 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:11:10.491764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.491696 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhvn4\"" Jul 14 18:11:10.491764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.491745 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:11:10.491982 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.491779 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:11:10.503877 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.503855 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4jp2"] Jul 14 18:11:10.505919 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.505903 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8st4g"] Jul 14 18:11:10.567025 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.566825 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fe26db-4700-4c31-9f83-219402644385-cert\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.567192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvrv\" (UniqueName: \"kubernetes.io/projected/27fe26db-4700-4c31-9f83-219402644385-kube-api-access-srvrv\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.567192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.567192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-crio-socket\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.567192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-data-volume\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.567365 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567216 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.567365 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.567246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fln\" (UniqueName: \"kubernetes.io/projected/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-api-access-h2fln\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.599417 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.599380 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jllgd"] Jul 14 18:11:10.602332 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.602307 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.604805 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:10.604767 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-default-jllgd\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="75db477c-1c43-443c-b5cc-3ce65f8e89a0" pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.604805 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:10.604786 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" Jul 14 18:11:10.605030 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:10.604788 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-zklb7\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zklb7\"" type="*v1.Secret" Jul 14 18:11:10.605030 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:10.604825 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" Jul 14 18:11:10.629229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.629192 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jllgd"] Jul 14 18:11:10.668210 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668175 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fe26db-4700-4c31-9f83-219402644385-cert\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvrv\" (UniqueName: \"kubernetes.io/projected/27fe26db-4700-4c31-9f83-219402644385-kube-api-access-srvrv\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-crio-socket\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-data-volume\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fln\" (UniqueName: \"kubernetes.io/projected/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-api-access-h2fln\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668679 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668563 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-crio-socket\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.668975 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.668950 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-data-volume\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.669188 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.669161 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.673753 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.673688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.673753 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.673725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fe26db-4700-4c31-9f83-219402644385-cert\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.681312 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.681282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fln\" (UniqueName: \"kubernetes.io/projected/22bab0bb-fb05-4ea8-9ee0-8e57a0958075-kube-api-access-h2fln\") pod \"insights-runtime-extractor-s4jp2\" (UID: \"22bab0bb-fb05-4ea8-9ee0-8e57a0958075\") " pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.681432 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.681400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvrv\" (UniqueName: \"kubernetes.io/projected/27fe26db-4700-4c31-9f83-219402644385-kube-api-access-srvrv\") pod \"ingress-canary-8st4g\" (UID: \"27fe26db-4700-4c31-9f83-219402644385\") " pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.769535 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.769488 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tss5t\" (UniqueName: \"kubernetes.io/projected/75db477c-1c43-443c-b5cc-3ce65f8e89a0-kube-api-access-tss5t\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.769535 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.769542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75db477c-1c43-443c-b5cc-3ce65f8e89a0-metrics-tls\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.769784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.769617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75db477c-1c43-443c-b5cc-3ce65f8e89a0-tmp-dir\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.769784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.769639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.787430 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.787395 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:10.799200 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.799168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4jp2" Jul 14 18:11:10.806090 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.806063 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8st4g" Jul 14 18:11:10.858897 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.858862 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:10.870260 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.870219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.870457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.870309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss5t\" (UniqueName: \"kubernetes.io/projected/75db477c-1c43-443c-b5cc-3ce65f8e89a0-kube-api-access-tss5t\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.870457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.870347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75db477c-1c43-443c-b5cc-3ce65f8e89a0-metrics-tls\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.870457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.870417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75db477c-1c43-443c-b5cc-3ce65f8e89a0-tmp-dir\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.870773 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.870751 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75db477c-1c43-443c-b5cc-3ce65f8e89a0-tmp-dir\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.879920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.879893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss5t\" (UniqueName: \"kubernetes.io/projected/75db477c-1c43-443c-b5cc-3ce65f8e89a0-kube-api-access-tss5t\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:10.924951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.924868 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:10.924951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.924892 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:10.925219 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.925078 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:10.928269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928082 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:11:10.928269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928095 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:11:10.928269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928137 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:11:10.928269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928221 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:11:10.928269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928230 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:11:10.928577 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:10.928497 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:11:11.072714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.072671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:11.075298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.075270 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0e4e0ae-937a-401d-a47c-6691c6926b23-original-pull-secret\") pod \"global-pull-secret-syncer-flxqs\" (UID: \"e0e4e0ae-937a-401d-a47c-6691c6926b23\") " pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:11.244775 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.244738 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flxqs" Jul 14 18:11:11.304312 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.304242 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:11.306764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.306732 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.309625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.309440 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 18:11:11.309625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.309459 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 18:11:11.309625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.309459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 18:11:11.309625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.309493 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5ljq\"" Jul 14 18:11:11.309938 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.309927 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 18:11:11.310238 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.310206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 18:11:11.310369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.310273 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 18:11:11.310439 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.310391 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 18:11:11.317644 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.317622 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 18:11:11.319152 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.319123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:11.375430 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375430 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssbx\" (UniqueName: \"kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375612 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.375837 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.375683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.476596 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476554 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssbx\" (UniqueName: \"kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.476844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477574 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.477540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477658 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.477584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477658 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.477627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.477775 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.477749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.479351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.479331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.479569 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.479553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.486576 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.486552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssbx\" (UniqueName: \"kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx\") pod \"console-9db7c5476-q7zcs\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.550053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.549978 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zklb7\"" Jul 14 18:11:11.619184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.619145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:11.703789 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.703759 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:11:11.713083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.713056 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75db477c-1c43-443c-b5cc-3ce65f8e89a0-metrics-tls\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:11.870903 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:11.870829 2567 configmap.go:193] Couldn't get configMap openshift-dns/dns-default: failed to sync configmap cache: timed out waiting for the condition Jul 14 18:11:11.871071 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:11.870964 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume podName:75db477c-1c43-443c-b5cc-3ce65f8e89a0 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:12.370945613 +0000 UTC m=+37.756844047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume") pod "dns-default-jllgd" (UID: "75db477c-1c43-443c-b5cc-3ce65f8e89a0") : failed to sync configmap cache: timed out waiting for the condition Jul 14 18:11:11.912240 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:11.912204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:12.001213 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.001188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:11:12.011324 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.011303 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:12.222206 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.222011 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8st4g"] Jul 14 18:11:12.224866 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:12.224838 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fe26db_4700_4c31_9f83_219402644385.slice/crio-3d6597e191729148919a73c4eae4afe57cbbe32cf730de0da70f47a723d2daaa WatchSource:0}: Error finding container 3d6597e191729148919a73c4eae4afe57cbbe32cf730de0da70f47a723d2daaa: Status 404 returned error can't find the container with id 3d6597e191729148919a73c4eae4afe57cbbe32cf730de0da70f47a723d2daaa Jul 14 18:11:12.230555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.230532 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:12.233437 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.233413 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flxqs"] Jul 14 18:11:12.233848 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.233828 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4jp2"] Jul 14 18:11:12.237784 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:12.237760 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d0bbd1_1783_40b5_9cfb_8e4f7ebd3ffe.slice/crio-1adae3ebd671fe1482f914f98a94af94eb2d68ed634d7096f384f05e7af9dedc WatchSource:0}: Error finding container 1adae3ebd671fe1482f914f98a94af94eb2d68ed634d7096f384f05e7af9dedc: Status 404 returned error can't find the container with id 1adae3ebd671fe1482f914f98a94af94eb2d68ed634d7096f384f05e7af9dedc Jul 14 18:11:12.238732 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:12.238706 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e4e0ae_937a_401d_a47c_6691c6926b23.slice/crio-4611f2b4202d195fe398686293d905c8d700280ae819449ed4bc2e1c3d860d78 WatchSource:0}: Error finding container 4611f2b4202d195fe398686293d905c8d700280ae819449ed4bc2e1c3d860d78: Status 404 returned error can't find the container with id 4611f2b4202d195fe398686293d905c8d700280ae819449ed4bc2e1c3d860d78 Jul 14 18:11:12.239555 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:12.239523 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bab0bb_fb05_4ea8_9ee0_8e57a0958075.slice/crio-1da5dcfae8897a87541eb1dafef3560df31eb1a39818fedba5982cf2bdb3c44b WatchSource:0}: Error finding container 1da5dcfae8897a87541eb1dafef3560df31eb1a39818fedba5982cf2bdb3c44b: Status 404 returned error can't find the container with id 1da5dcfae8897a87541eb1dafef3560df31eb1a39818fedba5982cf2bdb3c44b Jul 14 18:11:12.391377 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.391331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:12.391961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.391938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75db477c-1c43-443c-b5cc-3ce65f8e89a0-config-volume\") pod \"dns-default-jllgd\" (UID: \"75db477c-1c43-443c-b5cc-3ce65f8e89a0\") " pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:12.412537 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.412475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:12.547509 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.547373 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jllgd"] Jul 14 18:11:12.549770 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:12.549743 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75db477c_1c43_443c_b5cc_3ce65f8e89a0.slice/crio-f33e2f937f62b62a535c4d6322d50f83e55643657b770f264af6f310007c79bd WatchSource:0}: Error finding container f33e2f937f62b62a535c4d6322d50f83e55643657b770f264af6f310007c79bd: Status 404 returned error can't find the container with id f33e2f937f62b62a535c4d6322d50f83e55643657b770f264af6f310007c79bd Jul 14 18:11:12.747231 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.747197 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:12.846152 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:12.846057 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:13.117009 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.116915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllgd" event={"ID":"75db477c-1c43-443c-b5cc-3ce65f8e89a0","Type":"ContainerStarted","Data":"f33e2f937f62b62a535c4d6322d50f83e55643657b770f264af6f310007c79bd"} Jul 14 18:11:13.125705 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.124796 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="e9b1c63de5750906b7892610ffef2409e3ae4e976bf13b236e524a4e6bb21560" exitCode=0 Jul 14 18:11:13.125705 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.124904 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"e9b1c63de5750906b7892610ffef2409e3ae4e976bf13b236e524a4e6bb21560"} Jul 14 18:11:13.128180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.128114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9db7c5476-q7zcs" event={"ID":"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe","Type":"ContainerStarted","Data":"1adae3ebd671fe1482f914f98a94af94eb2d68ed634d7096f384f05e7af9dedc"} Jul 14 18:11:13.130020 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.129968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flxqs" event={"ID":"e0e4e0ae-937a-401d-a47c-6691c6926b23","Type":"ContainerStarted","Data":"4611f2b4202d195fe398686293d905c8d700280ae819449ed4bc2e1c3d860d78"} Jul 14 18:11:13.138128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.135376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4jp2" event={"ID":"22bab0bb-fb05-4ea8-9ee0-8e57a0958075","Type":"ContainerStarted","Data":"1da5dcfae8897a87541eb1dafef3560df31eb1a39818fedba5982cf2bdb3c44b"} Jul 14 18:11:13.142290 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.142234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8st4g" event={"ID":"27fe26db-4700-4c31-9f83-219402644385","Type":"ContainerStarted","Data":"3d6597e191729148919a73c4eae4afe57cbbe32cf730de0da70f47a723d2daaa"} Jul 14 18:11:13.401411 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.401319 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:13.477182 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.477149 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6ddtx"] Jul 14 18:11:13.487408 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.487332 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:13.488508 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.488488 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.495512 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.495780 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.496043 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.496314 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.496502 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:11:13.496871 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.496700 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lcn22\"" Jul 14 18:11:13.498386 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.497896 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6ddtx"] Jul 14 18:11:13.602842 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.602785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.603310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.602884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.603310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.603043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrvs\" (UniqueName: \"kubernetes.io/projected/95ac1ce4-aaa4-4834-817a-8f0f877450bd-kube-api-access-tgrvs\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.603310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.603085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac1ce4-aaa4-4834-817a-8f0f877450bd-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.705801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.703430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrvs\" (UniqueName: \"kubernetes.io/projected/95ac1ce4-aaa4-4834-817a-8f0f877450bd-kube-api-access-tgrvs\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.705801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.703509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac1ce4-aaa4-4834-817a-8f0f877450bd-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.705801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.703585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.705801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.703634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.705801 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.704725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac1ce4-aaa4-4834-817a-8f0f877450bd-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.706591 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:13.706295 2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 14 18:11:13.706591 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:13.706384 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls podName:95ac1ce4-aaa4-4834-817a-8f0f877450bd nodeName:}" failed. No retries permitted until 2026-07-14 18:11:14.206361019 +0000 UTC m=+39.592259465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls") pod "prometheus-operator-64785b6675-6ddtx" (UID: "95ac1ce4-aaa4-4834-817a-8f0f877450bd") : secret "prometheus-operator-tls" not found Jul 14 18:11:13.719383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.719335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.757529 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.757467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrvs\" (UniqueName: \"kubernetes.io/projected/95ac1ce4-aaa4-4834-817a-8f0f877450bd-kube-api-access-tgrvs\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:13.875634 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.875563 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:13.963689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:13.963533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:14.160081 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.159121 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea73bc14-42d0-4afd-98b5-8163d7ed0a68" containerID="cfc5c370bef58e8599f2d200454793f8d72f6169538fd1c26de2c3a146403bf3" exitCode=0 Jul 14 18:11:14.160081 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.159202 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerDied","Data":"cfc5c370bef58e8599f2d200454793f8d72f6169538fd1c26de2c3a146403bf3"} Jul 14 18:11:14.211516 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.211324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:14.215721 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.215657 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95ac1ce4-aaa4-4834-817a-8f0f877450bd-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6ddtx\" (UID: \"95ac1ce4-aaa4-4834-817a-8f0f877450bd\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:14.405730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.405456 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" Jul 14 18:11:14.614492 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:14.614067 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6ddtx"] Jul 14 18:11:14.618945 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:14.618909 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ac1ce4_aaa4_4834_817a_8f0f877450bd.slice/crio-320f37b5584842fdb293cf3cdd2fdf03d44f59fc10e7dd0f234daf3d9204bacd WatchSource:0}: Error finding container 320f37b5584842fdb293cf3cdd2fdf03d44f59fc10e7dd0f234daf3d9204bacd: Status 404 returned error can't find the container with id 320f37b5584842fdb293cf3cdd2fdf03d44f59fc10e7dd0f234daf3d9204bacd Jul 14 18:11:15.171693 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:15.170838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66tqb" event={"ID":"ea73bc14-42d0-4afd-98b5-8163d7ed0a68","Type":"ContainerStarted","Data":"b209c853e360684a0369dd81ba73c04e4ee75f4f6d93996b566b96e5eaee341f"} Jul 14 18:11:15.173356 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:15.173280 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" event={"ID":"95ac1ce4-aaa4-4834-817a-8f0f877450bd","Type":"ContainerStarted","Data":"320f37b5584842fdb293cf3cdd2fdf03d44f59fc10e7dd0f234daf3d9204bacd"} Jul 14 18:11:15.199773 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:15.199509 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-66tqb" podStartSLOduration=8.317782605 podStartE2EDuration="39.199486099s" totalImagesPullingTime="30.881703494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:15.197424095 +0000 UTC m=+40.583322551" watchObservedRunningTime="2026-07-14 18:11:15.199486099 +0000 UTC m=+40.585384555" Jul 14 18:11:15.327689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:15.327657 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:15.425904 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:15.425823 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:16.290434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:16.290146 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:16.368771 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:16.368729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:18.148989 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:18.148948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:18.249470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:18.249432 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:18.347566 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:18.347532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:19.957602 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:19.957564 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:20.032386 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:20.032345 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:23.939782 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:23.939706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:24.202105 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.202018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9db7c5476-q7zcs" event={"ID":"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe","Type":"ContainerStarted","Data":"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f"} Jul 14 18:11:24.203544 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.203511 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flxqs" event={"ID":"e0e4e0ae-937a-401d-a47c-6691c6926b23","Type":"ContainerStarted","Data":"72edf0ed1cab3c36c3c043eb8300d7150ca56370f7528a34c0bc7cb0d084a52a"} Jul 14 18:11:24.205223 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.205194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" event={"ID":"95ac1ce4-aaa4-4834-817a-8f0f877450bd","Type":"ContainerStarted","Data":"ea3f2077fdd73619d42752f0d6ce8de91581801daf8be7e3994577b4f4752166"} Jul 14 18:11:24.205351 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.205234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" event={"ID":"95ac1ce4-aaa4-4834-817a-8f0f877450bd","Type":"ContainerStarted","Data":"8f0fc9e58f1534ca3a128b695e759833803217e036e5fa40ea609f5f9eef9c65"} Jul 14 18:11:24.206512 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.206491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4jp2" event={"ID":"22bab0bb-fb05-4ea8-9ee0-8e57a0958075","Type":"ContainerStarted","Data":"a6e2c11fc5d08fdc6d075e585ad91b28733be553b5b8381249b7cbe80b776074"} Jul 14 18:11:24.207899 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.207875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8st4g" event={"ID":"27fe26db-4700-4c31-9f83-219402644385","Type":"ContainerStarted","Data":"4ce73625fa77f043120ff0878986a4b8e1399e713e830f97a483c477c0cee191"} Jul 14 18:11:24.209427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.209406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllgd" event={"ID":"75db477c-1c43-443c-b5cc-3ce65f8e89a0","Type":"ContainerStarted","Data":"8b2620b75820180a8bb91d373c17762c229ffffb67ceac8824e2bc2abfd4c2de"} Jul 14 18:11:24.209520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.209441 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllgd" event={"ID":"75db477c-1c43-443c-b5cc-3ce65f8e89a0","Type":"ContainerStarted","Data":"67c5923dfcdd36254c2fd8a60d59f151fa31e25fae38db6c36727807178637f6"} Jul 14 18:11:24.209520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.209468 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:24.235740 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.235688 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9db7c5476-q7zcs" podStartSLOduration=3.179568984 podStartE2EDuration="13.235667954s" totalImagesPullingTime="10.05609897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:24.234851531 +0000 UTC m=+49.620749975" watchObservedRunningTime="2026-07-14 18:11:24.235667954 +0000 UTC m=+49.621566401" Jul 14 18:11:24.254029 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.253943 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-flxqs" podStartSLOduration=10.062592954 podStartE2EDuration="21.253929156s" totalImagesPullingTime="11.191336202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:24.253690054 +0000 UTC m=+49.639588508" watchObservedRunningTime="2026-07-14 18:11:24.253929156 +0000 UTC m=+49.639827611" Jul 14 18:11:24.275226 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.275171 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jllgd" podStartSLOduration=4.530505896 podStartE2EDuration="14.275153488s" totalImagesPullingTime="9.744647592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:24.273796892 +0000 UTC m=+49.659695345" watchObservedRunningTime="2026-07-14 18:11:24.275153488 +0000 UTC m=+49.661051919" Jul 14 18:11:24.293001 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.292919 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-6ddtx" podStartSLOduration=2.4610320310000002 podStartE2EDuration="11.292897651s" totalImagesPullingTime="8.83186562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:24.292651467 +0000 UTC m=+49.678549921" watchObservedRunningTime="2026-07-14 18:11:24.292897651 +0000 UTC m=+49.678796104" Jul 14 18:11:24.312550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:24.312488 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8st4g" podStartSLOduration=3.107315915 podStartE2EDuration="14.312471694s" totalImagesPullingTime="11.205155779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:24.311716167 +0000 UTC m=+49.697614620" watchObservedRunningTime="2026-07-14 18:11:24.312471694 +0000 UTC m=+49.698370147" Jul 14 18:11:25.215251 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.215217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4jp2" event={"ID":"22bab0bb-fb05-4ea8-9ee0-8e57a0958075","Type":"ContainerStarted","Data":"788c1c61ca214fd1fb34d57c6fd5e51bded4510534109aeb9f977072d93e6f57"} Jul 14 18:11:25.250352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.250317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.310803 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.310773 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.422100 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.422060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.535021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.534970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.624390 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.624356 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.695783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.695748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.797698 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.797622 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.835770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.835737 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8"] Jul 14 18:11:25.847749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.847723 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:25.849876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.849847 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z6rv5\"" Jul 14 18:11:25.849876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.849872 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 18:11:25.850048 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.849847 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 18:11:25.855083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.855060 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t2d8l"] Jul 14 18:11:25.872191 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.872165 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8"] Jul 14 18:11:25.872353 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.872220 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xlqh7"] Jul 14 18:11:25.872427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.872373 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.875025 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.874996 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:11:25.875221 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.875196 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:11:25.875336 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.875314 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pj49j\"" Jul 14 18:11:25.876007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.875582 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:11:25.884689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.884661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:25.885642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.885622 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xlqh7"] Jul 14 18:11:25.885765 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.885745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.888414 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.888374 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 18:11:25.888609 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.888581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8qphl\"" Jul 14 18:11:25.888759 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.888743 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 18:11:25.888980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.888964 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 18:11:25.908347 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908479 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908362 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.908479 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908479 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908444 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-wtmp\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-sys\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908572 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-textfile\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdp2g\" (UniqueName: \"kubernetes.io/projected/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-kube-api-access-rdp2g\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-root\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908700 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.908875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a6b7512-3cbe-4300-ae4f-211c1809c6be-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.909168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.909168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-metrics-client-ca\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.909168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.908989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94tx\" (UniqueName: \"kubernetes.io/projected/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-kube-api-access-p94tx\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:25.909168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.909034 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:25.909168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.909067 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lgq\" (UniqueName: \"kubernetes.io/projected/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-api-access-v7lgq\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:25.964579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:25.964546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.009592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a6b7512-3cbe-4300-ae4f-211c1809c6be-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.009592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.009846 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-metrics-client-ca\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.009846 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009644 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94tx\" (UniqueName: \"kubernetes.io/projected/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-kube-api-access-p94tx\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.009846 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.009846 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.009701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lgq\" (UniqueName: \"kubernetes.io/projected/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-api-access-v7lgq\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.010066 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010030 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3a6b7512-3cbe-4300-ae4f-211c1809c6be-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.010121 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010174 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.010174 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010273 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010273 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-wtmp\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-sys\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010337 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-textfile\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.010520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdp2g\" (UniqueName: \"kubernetes.io/projected/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-kube-api-access-rdp2g\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.010520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-root\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.010520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.010716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-metrics-client-ca\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.010716 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:26.010615 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 14 18:11:26.010716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-wtmp\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010716 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:26.010679 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls podName:b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe nodeName:}" failed. No retries permitted until 2026-07-14 18:11:26.510660994 +0000 UTC m=+51.896559439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-qkwf8" (UID: "b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe") : secret "openshift-state-metrics-tls" not found Jul 14 18:11:26.010985 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010796 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-sys\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.010985 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.010956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-root\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.011085 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.011068 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-textfile\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.011740 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:26.011640 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 18:11:26.011740 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.011662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.011740 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.011689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.011740 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:26.011708 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls podName:dd053b0f-e96c-4e0a-8669-1f2b2c3ff499 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:26.5116912 +0000 UTC m=+51.897589631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls") pod "node-exporter-t2d8l" (UID: "dd053b0f-e96c-4e0a-8669-1f2b2c3ff499") : secret "node-exporter-tls" not found Jul 14 18:11:26.012042 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.011857 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.012042 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.011884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.013711 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.013668 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.013711 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.013691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.013881 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.013691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.015388 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.015369 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.022995 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.022962 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdp2g\" (UniqueName: \"kubernetes.io/projected/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-kube-api-access-rdp2g\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.023228 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.023204 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lgq\" (UniqueName: \"kubernetes.io/projected/3a6b7512-3cbe-4300-ae4f-211c1809c6be-kube-api-access-v7lgq\") pod \"kube-state-metrics-785778ff74-xlqh7\" (UID: \"3a6b7512-3cbe-4300-ae4f-211c1809c6be\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.023478 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.023457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94tx\" (UniqueName: \"kubernetes.io/projected/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-kube-api-access-p94tx\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.069738 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.069658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.177046 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.177014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.196109 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.196079 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" Jul 14 18:11:26.279776 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.279739 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.516497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.516466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.516714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.516524 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.519144 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.519116 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd053b0f-e96c-4e0a-8669-1f2b2c3ff499-node-exporter-tls\") pod \"node-exporter-t2d8l\" (UID: \"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499\") " pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.519271 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.519179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qkwf8\" (UID: \"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.566542 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.566501 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.656435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.656398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.759670 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.759636 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" Jul 14 18:11:26.785922 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.785825 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2d8l" Jul 14 18:11:26.839605 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.839569 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.959411 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.959383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:26.994033 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:26.994005 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:11:27.052580 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.052497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:11:27.052751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.052636 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.058010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.057983 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 18:11:27.058149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.057990 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 18:11:27.058208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058172 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 18:11:27.058208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 18:11:27.058304 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 18:11:27.058353 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058300 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 18:11:27.058353 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058329 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 18:11:27.058497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058478 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8tm55\"" Jul 14 18:11:27.058497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058494 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 18:11:27.058730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.058710 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 18:11:27.123553 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.123513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.123757 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.123557 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-config-out\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.124025 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.124002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.124104 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.124051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-web-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.124104 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.124093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.124232 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.124199 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjph9\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-kube-api-access-gjph9\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128488 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.124276 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-config-volume\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128651 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.128853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.128845 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-config-volume\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230175 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230664 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-config-out\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230664 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230664 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-web-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230664 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230430 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230664 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjph9\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-kube-api-access-gjph9\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230935 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.230839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.230985 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:27.230948 2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 14 18:11:27.231037 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:27.230993 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls podName:c77b60a3-5ced-41a9-95e9-c24a12203274 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:27.730979647 +0000 UTC m=+53.116878092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c77b60a3-5ced-41a9-95e9-c24a12203274") : secret "alertmanager-main-tls" not found Jul 14 18:11:27.232499 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.231264 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.232499 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.231944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77b60a3-5ced-41a9-95e9-c24a12203274-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.235129 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.235025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.235834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.235781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.236490 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.236447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-config-volume\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.236573 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.236497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.236999 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.236977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.237255 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.237227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-web-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.237559 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.237536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c77b60a3-5ced-41a9-95e9-c24a12203274-config-out\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.237649 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.237592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.240216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.240174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjph9\" (UniqueName: \"kubernetes.io/projected/c77b60a3-5ced-41a9-95e9-c24a12203274-kube-api-access-gjph9\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.394034 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:27.393997 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd053b0f_e96c_4e0a_8669_1f2b2c3ff499.slice/crio-bde50a321f1fbb066c3bf5e3d2cb3e65ba0cf8c1c4f67937be0970cd12cf57d2 WatchSource:0}: Error finding container bde50a321f1fbb066c3bf5e3d2cb3e65ba0cf8c1c4f67937be0970cd12cf57d2: Status 404 returned error can't find the container with id bde50a321f1fbb066c3bf5e3d2cb3e65ba0cf8c1c4f67937be0970cd12cf57d2 Jul 14 18:11:27.536690 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.536659 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8"] Jul 14 18:11:27.541011 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:27.540983 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7832a17_4a68_4cf7_a8ff_8dd90fd9fcbe.slice/crio-75d65845ede40e03946780494c3f704425eafcdea9254cab1fd167e403ce30bc WatchSource:0}: Error finding container 75d65845ede40e03946780494c3f704425eafcdea9254cab1fd167e403ce30bc: Status 404 returned error can't find the container with id 75d65845ede40e03946780494c3f704425eafcdea9254cab1fd167e403ce30bc Jul 14 18:11:27.568706 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.568556 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xlqh7"] Jul 14 18:11:27.620673 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.620643 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:27.711290 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:27.711254 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6b7512_3cbe_4300_ae4f_211c1809c6be.slice/crio-9fa4b3ec4b31ed481aab419c431596554682c18b236b7420ce930bcfd4b9e30d WatchSource:0}: Error finding container 9fa4b3ec4b31ed481aab419c431596554682c18b236b7420ce930bcfd4b9e30d: Status 404 returned error can't find the container with id 9fa4b3ec4b31ed481aab419c431596554682c18b236b7420ce930bcfd4b9e30d Jul 14 18:11:27.734259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.734235 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.736957 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.736928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c77b60a3-5ced-41a9-95e9-c24a12203274-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c77b60a3-5ced-41a9-95e9-c24a12203274\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.965924 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.965807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:11:27.968114 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.968086 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84457cb55-bgwd5"] Jul 14 18:11:27.991173 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.991137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84457cb55-bgwd5"] Jul 14 18:11:27.991344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.991221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:27.993700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.993674 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 14 18:11:27.993854 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.993719 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 14 18:11:27.993854 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.993745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 14 18:11:27.994125 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.994111 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b4b4ii71q21d6\"" Jul 14 18:11:27.994199 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.994183 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 14 18:11:27.994378 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.994362 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fk2fm\"" Jul 14 18:11:27.994473 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:27.994368 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 14 18:11:28.036760 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.036720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.036932 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.036891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037034 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.036956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037034 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.037016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.037066 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-metrics-client-ca\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.037100 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037247 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.037161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4przh\" (UniqueName: \"kubernetes.io/projected/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-kube-api-access-4przh\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.037247 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.037202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-grpc-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.129041 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.129018 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:11:28.129712 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.129687 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:28.130731 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:28.130710 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77b60a3_5ced_41a9_95e9_c24a12203274.slice/crio-1c6db72236fa1130007c83cf0f0d796564fdcd7750bc3bd7c1445bdacbfa0baf WatchSource:0}: Error finding container 1c6db72236fa1130007c83cf0f0d796564fdcd7750bc3bd7c1445bdacbfa0baf: Status 404 returned error can't find the container with id 1c6db72236fa1130007c83cf0f0d796564fdcd7750bc3bd7c1445bdacbfa0baf Jul 14 18:11:28.138377 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138353 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138385 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-metrics-client-ca\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138608 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138608 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4przh\" (UniqueName: \"kubernetes.io/projected/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-kube-api-access-4przh\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138608 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-grpc-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.138733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.138649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.139252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.139217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-metrics-client-ca\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.140912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.140881 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.141186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.141162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-grpc-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.141233 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.141191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.141366 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.141346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.141437 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.141340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.141437 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.141381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-secret-thanos-querier-tls\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.148094 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.148076 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4przh\" (UniqueName: \"kubernetes.io/projected/b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1-kube-api-access-4przh\") pod \"thanos-querier-84457cb55-bgwd5\" (UID: \"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1\") " pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.230058 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.230021 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:28.232372 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.232337 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4jp2" event={"ID":"22bab0bb-fb05-4ea8-9ee0-8e57a0958075","Type":"ContainerStarted","Data":"afa3e866241fe4f8ae119528b07409050c4b45c7cebb3de78d9a2f9ab1b61848"} Jul 14 18:11:28.233710 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.233685 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"1c6db72236fa1130007c83cf0f0d796564fdcd7750bc3bd7c1445bdacbfa0baf"} Jul 14 18:11:28.235045 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.235019 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2d8l" event={"ID":"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499","Type":"ContainerStarted","Data":"bde50a321f1fbb066c3bf5e3d2cb3e65ba0cf8c1c4f67937be0970cd12cf57d2"} Jul 14 18:11:28.236421 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.236396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" event={"ID":"3a6b7512-3cbe-4300-ae4f-211c1809c6be","Type":"ContainerStarted","Data":"9fa4b3ec4b31ed481aab419c431596554682c18b236b7420ce930bcfd4b9e30d"} Jul 14 18:11:28.238524 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.238501 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" event={"ID":"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe","Type":"ContainerStarted","Data":"c17811dc755763c0724ee58cb6cfd72ee1c474be8af3224dd65a27d40ce7780a"} Jul 14 18:11:28.238601 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.238555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" event={"ID":"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe","Type":"ContainerStarted","Data":"e9723bfa53f6e8b85bcec926bcc52e9b2f3179c69d9bc258b7e351432e9eb53e"} Jul 14 18:11:28.238601 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.238584 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" event={"ID":"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe","Type":"ContainerStarted","Data":"75d65845ede40e03946780494c3f704425eafcdea9254cab1fd167e403ce30bc"} Jul 14 18:11:28.253630 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.253570 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4jp2" podStartSLOduration=3.084292088 podStartE2EDuration="18.253550711s" totalImagesPullingTime="15.169258623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:11:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:28.252608519 +0000 UTC m=+53.638506971" watchObservedRunningTime="2026-07-14 18:11:28.253550711 +0000 UTC m=+53.639449164" Jul 14 18:11:28.300918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.300868 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:28.492337 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.492245 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84457cb55-bgwd5"] Jul 14 18:11:28.496342 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:28.496309 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c72ef5_78e9_4877_a5c5_758bfbcdfdb1.slice/crio-953087cb32255e76946cc5657a9ded8be546e0ee0b319599442766299c758ec5 WatchSource:0}: Error finding container 953087cb32255e76946cc5657a9ded8be546e0ee0b319599442766299c758ec5: Status 404 returned error can't find the container with id 953087cb32255e76946cc5657a9ded8be546e0ee0b319599442766299c758ec5 Jul 14 18:11:28.878209 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.878088 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:28.981075 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:28.981034 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:29.141883 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:29.141754 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:29.244609 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:29.244559 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"953087cb32255e76946cc5657a9ded8be546e0ee0b319599442766299c758ec5"} Jul 14 18:11:29.373186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:29.373141 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:30.235329 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.235286 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bcb994478-jgh2r"] Jul 14 18:11:30.255136 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.255103 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcb994478-jgh2r"] Jul 14 18:11:30.255269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.255234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.258487 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.258448 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 18:11:30.259021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.258780 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-21shj0c6qvucn\"" Jul 14 18:11:30.259021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.258792 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 18:11:30.259021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.258787 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 18:11:30.259021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.258369 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 18:11:30.260082 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.259249 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jdj45\"" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359366 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bz2\" (UniqueName: \"kubernetes.io/projected/d307dd2d-c841-4155-b8fb-2b67110dfdaa-kube-api-access-76bz2\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-metrics-server-audit-profiles\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-tls\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359582 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-client-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.359833 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.359694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d307dd2d-c841-4155-b8fb-2b67110dfdaa-audit-log\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bz2\" (UniqueName: \"kubernetes.io/projected/d307dd2d-c841-4155-b8fb-2b67110dfdaa-kube-api-access-76bz2\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-metrics-server-audit-profiles\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-tls\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.460834 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-client-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.461189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.460901 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d307dd2d-c841-4155-b8fb-2b67110dfdaa-audit-log\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.462179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.461322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d307dd2d-c841-4155-b8fb-2b67110dfdaa-audit-log\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.462179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.462063 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-metrics-server-audit-profiles\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.462449 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.462423 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d307dd2d-c841-4155-b8fb-2b67110dfdaa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.464141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.464113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-tls\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.464556 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.464516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-client-ca-bundle\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.465389 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.465371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d307dd2d-c841-4155-b8fb-2b67110dfdaa-secret-metrics-server-client-certs\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.474081 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.474061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bz2\" (UniqueName: \"kubernetes.io/projected/d307dd2d-c841-4155-b8fb-2b67110dfdaa-kube-api-access-76bz2\") pod \"metrics-server-6bcb994478-jgh2r\" (UID: \"d307dd2d-c841-4155-b8fb-2b67110dfdaa\") " pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.570837 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.567314 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:30.631073 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.630913 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d"] Jul 14 18:11:30.634334 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.634279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.634658 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.634606 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:30.638294 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.638268 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 18:11:30.638525 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.638499 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m8x6r\"" Jul 14 18:11:30.652642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.652600 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d"] Jul 14 18:11:30.664298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.663852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.664298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.663962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2nf\" (UniqueName: \"kubernetes.io/projected/eddae40e-6050-4fd9-a04e-bee4ef47ac55-kube-api-access-kk2nf\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.681097 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.681032 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:30.681252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.681117 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:11:30.683920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.683899 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.711409 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.711375 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:11:30.764605 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.764347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:30.765288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.764870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2nf\" (UniqueName: \"kubernetes.io/projected/eddae40e-6050-4fd9-a04e-bee4ef47ac55-kube-api-access-kk2nf\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.765288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.764955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.765288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.764984 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mmm\" (UniqueName: \"kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765783 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.765842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:30.766023 2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 14 18:11:30.766201 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:30.766080 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert podName:eddae40e-6050-4fd9-a04e-bee4ef47ac55 nodeName:}" failed. No retries permitted until 2026-07-14 18:11:31.266062215 +0000 UTC m=+56.651960648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-75s4d" (UID: "eddae40e-6050-4fd9-a04e-bee4ef47ac55") : secret "monitoring-plugin-cert" not found Jul 14 18:11:30.777193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.777143 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2nf\" (UniqueName: \"kubernetes.io/projected/eddae40e-6050-4fd9-a04e-bee4ef47ac55-kube-api-access-kk2nf\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:30.867170 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867170 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867270 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867306 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mmm\" (UniqueName: \"kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.867470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.867412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.868443 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.868398 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.868550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.868485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.869236 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.869195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.869392 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.869333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.870744 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.870693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.871182 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.871159 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.878369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.878343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mmm\" (UniqueName: \"kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm\") pod \"console-6c4b4644fd-mcwjq\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.992438 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.992401 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:30.998102 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.996489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:30.998923 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:30.998834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bcb994478-jgh2r"] Jul 14 18:11:31.031009 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:31.030969 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd307dd2d_c841_4155_b8fb_2b67110dfdaa.slice/crio-e0db7f189b9998f339c4d997cb69db2ef93942ff99effab2afcf2a4adcf5afdd WatchSource:0}: Error finding container e0db7f189b9998f339c4d997cb69db2ef93942ff99effab2afcf2a4adcf5afdd: Status 404 returned error can't find the container with id e0db7f189b9998f339c4d997cb69db2ef93942ff99effab2afcf2a4adcf5afdd Jul 14 18:11:31.143395 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.143306 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:11:31.146772 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:31.146737 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561a5fe8_0f84_4350_98eb_f0b9d2ebaaa8.slice/crio-c7bfdc96c8716a7a18fa68a01d3911e43254ca782c125d9d8cec878bd7d58595 WatchSource:0}: Error finding container c7bfdc96c8716a7a18fa68a01d3911e43254ca782c125d9d8cec878bd7d58595: Status 404 returned error can't find the container with id c7bfdc96c8716a7a18fa68a01d3911e43254ca782c125d9d8cec878bd7d58595 Jul 14 18:11:31.255926 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.255888 2567 generic.go:356] "Generic (PLEG): container finished" podID="c77b60a3-5ced-41a9-95e9-c24a12203274" containerID="e36e47ae9de05b82d3ebd2618720b3c87c0f680d0d927cac15f0c46ac1bb5157" exitCode=0 Jul 14 18:11:31.256357 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.255976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerDied","Data":"e36e47ae9de05b82d3ebd2618720b3c87c0f680d0d927cac15f0c46ac1bb5157"} Jul 14 18:11:31.258027 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.258003 2567 generic.go:356] "Generic (PLEG): container finished" podID="dd053b0f-e96c-4e0a-8669-1f2b2c3ff499" containerID="4256f3965ee77ef0792d7b92a70f3464cbd177e9e696c91debd4fe74bc487842" exitCode=0 Jul 14 18:11:31.258109 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.258093 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2d8l" event={"ID":"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499","Type":"ContainerDied","Data":"4256f3965ee77ef0792d7b92a70f3464cbd177e9e696c91debd4fe74bc487842"} Jul 14 18:11:31.260753 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.260723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" event={"ID":"3a6b7512-3cbe-4300-ae4f-211c1809c6be","Type":"ContainerStarted","Data":"3f7bce3cb53a49f6e4163f234e0f0542bd2834109436e790efb9e2a6a3ed19de"} Jul 14 18:11:31.260866 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.260785 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" event={"ID":"3a6b7512-3cbe-4300-ae4f-211c1809c6be","Type":"ContainerStarted","Data":"b4a70aebe814d3218cd567e580f48225dbaab52ecbf16ce3329566abd7562657"} Jul 14 18:11:31.260866 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.260840 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" event={"ID":"3a6b7512-3cbe-4300-ae4f-211c1809c6be","Type":"ContainerStarted","Data":"c281a2011b3d44d468360e339efcaef21f662e9e9e9eb2399d1c7b8c1f46557c"} Jul 14 18:11:31.264577 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.264552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" event={"ID":"b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe","Type":"ContainerStarted","Data":"d3b1977106801ae30e5fd1369d744f5c5a4099a2f7089e51d7c2f359e782a11a"} Jul 14 18:11:31.266243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.266216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" event={"ID":"d307dd2d-c841-4155-b8fb-2b67110dfdaa","Type":"ContainerStarted","Data":"e0db7f189b9998f339c4d997cb69db2ef93942ff99effab2afcf2a4adcf5afdd"} Jul 14 18:11:31.268059 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.268036 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b4644fd-mcwjq" event={"ID":"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8","Type":"ContainerStarted","Data":"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13"} Jul 14 18:11:31.268154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.268076 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b4644fd-mcwjq" event={"ID":"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8","Type":"ContainerStarted","Data":"c7bfdc96c8716a7a18fa68a01d3911e43254ca782c125d9d8cec878bd7d58595"} Jul 14 18:11:31.271823 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.271783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:31.274604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.274581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eddae40e-6050-4fd9-a04e-bee4ef47ac55-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-75s4d\" (UID: \"eddae40e-6050-4fd9-a04e-bee4ef47ac55\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:31.314307 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.314241 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qkwf8" podStartSLOduration=4.107713203 podStartE2EDuration="6.314218744s" totalImagesPullingTime="2.206505541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:31.313747927 +0000 UTC m=+56.699646382" watchObservedRunningTime="2026-07-14 18:11:31.314218744 +0000 UTC m=+56.700117198" Jul 14 18:11:31.340639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.340582 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-xlqh7" podStartSLOduration=3.9273329390000002 podStartE2EDuration="6.340563413s" totalImagesPullingTime="2.413230474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:31.339166534 +0000 UTC m=+56.725064999" watchObservedRunningTime="2026-07-14 18:11:31.340563413 +0000 UTC m=+56.726461865" Jul 14 18:11:31.376332 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.376278 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4b4644fd-mcwjq" podStartSLOduration=1.376261386 podStartE2EDuration="1.376261386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:31.37498392 +0000 UTC m=+56.760882372" watchObservedRunningTime="2026-07-14 18:11:31.376261386 +0000 UTC m=+56.762159830" Jul 14 18:11:31.554957 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.554922 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:31.619786 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.619746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:31.976940 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:31.976857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d"] Jul 14 18:11:32.206520 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.206450 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:32.212320 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.212281 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:11:32.217885 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.217857 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.222060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.222035 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 18:11:32.222220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.222044 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 18:11:32.222781 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.222763 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1nvv8ke8nv1a9\"" Jul 14 18:11:32.222975 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.222891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 18:11:32.223591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.223533 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 18:11:32.223591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.223575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 18:11:32.223853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.223806 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 18:11:32.223947 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.223918 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5lm5\"" Jul 14 18:11:32.224128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.224082 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 18:11:32.224595 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.224575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 18:11:32.227005 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.226372 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 18:11:32.227005 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.226628 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 18:11:32.228862 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.228676 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 18:11:32.229242 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.229221 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 18:11:32.231570 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.231547 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 18:11:32.234177 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.234007 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 18:11:32.238380 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.238352 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:11:32.281226 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.281136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"95e5080027ec5f2a1022fcc7ff261b4140b06831a9a1f55351aff7a9e9537af6"} Jul 14 18:11:32.281226 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.281224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"b853cd13a38eedbc37a01f7808bba0b04a24b810c959572f1767c40ecfa08693"} Jul 14 18:11:32.281688 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.281267 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"21e8987d668f670390123efb8317977052d8837eb8f95b832630db88a00c6c6f"} Jul 14 18:11:32.282966 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.282937 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" event={"ID":"eddae40e-6050-4fd9-a04e-bee4ef47ac55","Type":"ContainerStarted","Data":"0eddeff08e6373fd99dcf4947d735854d92e0ad6262f4fd79e95ed85d57857c1"} Jul 14 18:11:32.284173 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94j4\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-kube-api-access-r94j4\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284398 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284397 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284560 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.284761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284779 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284837 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.284967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.285089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.285114 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.285130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.285263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.285155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.286640 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.286557 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2d8l" event={"ID":"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499","Type":"ContainerStarted","Data":"02a7c78f4300a81f1ed0ee227a971888d437b9d425c839b9bad4e88508aabc85"} Jul 14 18:11:32.286640 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.286608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2d8l" event={"ID":"dd053b0f-e96c-4e0a-8669-1f2b2c3ff499","Type":"ContainerStarted","Data":"cf7ae103aa9b46746f4cb3dab8cdd1df0c354bacae34157c35b99022161c50ad"} Jul 14 18:11:32.308696 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.308635 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t2d8l" podStartSLOduration=4.442938217 podStartE2EDuration="7.308610233s" totalImagesPullingTime="2.725160112s" totalInitContainerRuntime="140.511904ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:32.307062713 +0000 UTC m=+57.692961166" watchObservedRunningTime="2026-07-14 18:11:32.308610233 +0000 UTC m=+57.694508700" Jul 14 18:11:32.386827 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.386723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.387010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.386916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.387010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.386957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.387147 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387016 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.387147 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.387794 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94j4\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-kube-api-access-r94j4\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.388901 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.390292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.389953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.390246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.390633 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.390749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.391075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.391258 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.391529 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.391354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.393623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.392494 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.393623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.393284 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.393872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.394175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.394343 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.395354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.395988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.397355 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.396679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.398962 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.398934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.399110 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.399084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fec06e9f-7f74-4454-834a-3dfd0cc504e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.399863 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.399798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fec06e9f-7f74-4454-834a-3dfd0cc504e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.413041 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.413009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94j4\" (UniqueName: \"kubernetes.io/projected/fec06e9f-7f74-4454-834a-3dfd0cc504e7-kube-api-access-r94j4\") pod \"prometheus-k8s-0\" (UID: \"fec06e9f-7f74-4454-834a-3dfd0cc504e7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.532346 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.532303 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:32.718188 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:32.717725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:11:32.720205 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:32.720175 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec06e9f_7f74_4454_834a_3dfd0cc504e7.slice/crio-65dde85bf3c8f593f2ed2d97f1ee93bc82e47c71d55e67c90dcf7c0ee03d9e60 WatchSource:0}: Error finding container 65dde85bf3c8f593f2ed2d97f1ee93bc82e47c71d55e67c90dcf7c0ee03d9e60: Status 404 returned error can't find the container with id 65dde85bf3c8f593f2ed2d97f1ee93bc82e47c71d55e67c90dcf7c0ee03d9e60 Jul 14 18:11:33.293062 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:33.293022 2567 generic.go:356] "Generic (PLEG): container finished" podID="fec06e9f-7f74-4454-834a-3dfd0cc504e7" containerID="78f7f4b53c2600d491b18626a384a88df53ff11c8f3cd7327f2c3c009a170253" exitCode=0 Jul 14 18:11:33.293454 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:33.293126 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerDied","Data":"78f7f4b53c2600d491b18626a384a88df53ff11c8f3cd7327f2c3c009a170253"} Jul 14 18:11:33.293454 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:33.293217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"65dde85bf3c8f593f2ed2d97f1ee93bc82e47c71d55e67c90dcf7c0ee03d9e60"} Jul 14 18:11:34.194716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.194653 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:34.218083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.218052 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jllgd" Jul 14 18:11:34.310859 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.307990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:34.316951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.316873 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"e47d0d1a7d46b851e3ec5b1f029b94aead78dead45ea58ec8817228e2b6cfe04"} Jul 14 18:11:34.317097 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.316991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"34c910a2b7ef294dc199bb23f817fe07ed379f06d70203fea99bcaab7d2181f2"} Jul 14 18:11:34.322121 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.322068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"5dc2aeda55b36ca02e3e8c20a31461b49049c2179a194e003d6a049ceb4f18a6"} Jul 14 18:11:34.322264 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.322235 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"510eafd3550631df94fb5bbe468b325069086a974c4b48246b2f4646082bd5ee"} Jul 14 18:11:34.324998 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.324360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" event={"ID":"d307dd2d-c841-4155-b8fb-2b67110dfdaa","Type":"ContainerStarted","Data":"08c42487d261aa0c578765dcc0745ed46a34e633eba6aa3b7acf9500ab8f970b"} Jul 14 18:11:34.327791 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.327767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" event={"ID":"eddae40e-6050-4fd9-a04e-bee4ef47ac55","Type":"ContainerStarted","Data":"27d8dbbfe8a515f569bb00071f83d6d1e1352f0b063b9f224cf325447b8b2bcd"} Jul 14 18:11:34.328442 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.328391 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:34.338389 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.338336 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" Jul 14 18:11:34.347962 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.347911 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" podStartSLOduration=1.25025566 podStartE2EDuration="4.347892465s" totalImagesPullingTime="3.097636805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:34.346187872 +0000 UTC m=+59.732086326" watchObservedRunningTime="2026-07-14 18:11:34.347892465 +0000 UTC m=+59.733790919" Jul 14 18:11:34.367775 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:34.367600 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-75s4d" podStartSLOduration=2.224601052 podStartE2EDuration="4.367580614s" totalImagesPullingTime="2.142979562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:34.366700025 +0000 UTC m=+59.752598478" watchObservedRunningTime="2026-07-14 18:11:34.367580614 +0000 UTC m=+59.753479051" Jul 14 18:11:35.088424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.088395 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmgf9" Jul 14 18:11:35.338197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.337964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"caecee60085d2c82f07adb9616d232212bc22e61dc47e6cca2ed22de4316aa80"} Jul 14 18:11:35.338197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.338080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"afb7c3502db637535139b998d5d237ea79a6ed185b5f32f97f0f0ced89369cf1"} Jul 14 18:11:35.338197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.338142 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"b517162bb7858c1c14364d0e2105ce96796e987997230823d76a9d5012921970"} Jul 14 18:11:35.338197 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.338144 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:35.338776 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.338286 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c77b60a3-5ced-41a9-95e9-c24a12203274","Type":"ContainerStarted","Data":"6b0b50f8bc5d548133c227162cb859284ba81a609133bcaac841aa57dfb440e5"} Jul 14 18:11:35.341615 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.341555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" event={"ID":"b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1","Type":"ContainerStarted","Data":"c4c0606f8144eaa8a1392042d4b5ec96d99dcda5e06290f759ab2dd2de11bf13"} Jul 14 18:11:35.382010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.381958 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.738154294 podStartE2EDuration="9.381940016s" totalImagesPullingTime="5.628150321s" totalInitContainerRuntime="15.635401ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:35.379455982 +0000 UTC m=+60.765354434" watchObservedRunningTime="2026-07-14 18:11:35.381940016 +0000 UTC m=+60.767838477" Jul 14 18:11:35.409146 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.409085 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" podStartSLOduration=2.999789758 podStartE2EDuration="8.40906197s" totalImagesPullingTime="5.409272212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:35.406749903 +0000 UTC m=+60.792648369" watchObservedRunningTime="2026-07-14 18:11:35.40906197 +0000 UTC m=+60.794960404" Jul 14 18:11:35.416723 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:35.416694 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:36.345523 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:36.345490 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:37.359984 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:37.359954 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84457cb55-bgwd5" Jul 14 18:11:38.358159 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358059 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"ea78f4e978b212e62e387c67f7f36b03e33dced3d3059a44ffbb6663b1c1da1a"} Jul 14 18:11:38.358159 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"d04804a624f3b4c82d1022f8c039e7b1e5718cc962f5a0202879371bcef6e275"} Jul 14 18:11:38.358159 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358161 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"8cbbdd64f544c0ec1edcd94fd79467e23d02f8ca7a0b3c82650a15ab5fa2f544"} Jul 14 18:11:38.358424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"9e8ed7d326d784b5edd79e71121ebd0512d3283e3268b620aacfa7c08b146dde"} Jul 14 18:11:38.358424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"e371589c71a4b6059e94f679b4d57add8e0483a60474172347812b4499a0fc77"} Jul 14 18:11:38.358424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.358233 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fec06e9f-7f74-4454-834a-3dfd0cc504e7","Type":"ContainerStarted","Data":"3b79c94915dd3c5a802057d77d5cc358c3cc0e25da9771aa6caea66b2bc6aa9b"} Jul 14 18:11:38.399868 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:38.399787 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.87803084 podStartE2EDuration="6.399765303s" totalImagesPullingTime="4.505293444s" totalInitContainerRuntime="16.441019ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:38.397599786 +0000 UTC m=+63.783498240" watchObservedRunningTime="2026-07-14 18:11:38.399765303 +0000 UTC m=+63.785663759" Jul 14 18:11:40.576534 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.576493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:40.579240 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.579221 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:11:40.589236 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.589204 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7848077b-6e8d-41bf-a3aa-7e8ec73be770-metrics-certs\") pod \"network-metrics-daemon-h6ww8\" (UID: \"7848077b-6e8d-41bf-a3aa-7e8ec73be770\") " pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:40.639478 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.639441 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jkhfz\"" Jul 14 18:11:40.647282 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.647257 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6ww8" Jul 14 18:11:40.775876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.775842 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6ww8"] Jul 14 18:11:40.778554 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.778531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:40.779022 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:40.778999 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7848077b_6e8d_41bf_a3aa_7e8ec73be770.slice/crio-15a6144d3042a4ecfb1dcc9a9a70a175825ad82ce327f1c0ea48f52a23ec3c42 WatchSource:0}: Error finding container 15a6144d3042a4ecfb1dcc9a9a70a175825ad82ce327f1c0ea48f52a23ec3c42: Status 404 returned error can't find the container with id 15a6144d3042a4ecfb1dcc9a9a70a175825ad82ce327f1c0ea48f52a23ec3c42 Jul 14 18:11:40.782227 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.782210 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:11:40.791408 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.791380 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:11:40.801784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.801761 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpqs\" (UniqueName: \"kubernetes.io/projected/4aed1a3f-e696-4a12-ab89-ce7a961c4684-kube-api-access-6xpqs\") pod \"network-check-target-5ksdk\" (UID: \"4aed1a3f-e696-4a12-ab89-ce7a961c4684\") " pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:40.952467 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.952391 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5vgxh\"" Jul 14 18:11:40.960252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.960223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:40.997566 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.997446 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:40.997566 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:40.997526 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:41.003016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.002988 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:41.094402 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.094238 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5ksdk"] Jul 14 18:11:41.096998 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:11:41.096973 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aed1a3f_e696_4a12_ab89_ce7a961c4684.slice/crio-17189a21a4e1cbfab747da9d269d788dfcb2eca486fb24b78622b5c9bccbc416 WatchSource:0}: Error finding container 17189a21a4e1cbfab747da9d269d788dfcb2eca486fb24b78622b5c9bccbc416: Status 404 returned error can't find the container with id 17189a21a4e1cbfab747da9d269d788dfcb2eca486fb24b78622b5c9bccbc416 Jul 14 18:11:41.371233 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.371185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ksdk" event={"ID":"4aed1a3f-e696-4a12-ab89-ce7a961c4684","Type":"ContainerStarted","Data":"17189a21a4e1cbfab747da9d269d788dfcb2eca486fb24b78622b5c9bccbc416"} Jul 14 18:11:41.372353 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.372327 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6ww8" event={"ID":"7848077b-6e8d-41bf-a3aa-7e8ec73be770","Type":"ContainerStarted","Data":"15a6144d3042a4ecfb1dcc9a9a70a175825ad82ce327f1c0ea48f52a23ec3c42"} Jul 14 18:11:41.376194 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.376170 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:11:41.542258 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.542224 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:41.646566 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.646490 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:41.749777 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:41.749742 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:42.533147 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:42.533121 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:11:43.081997 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.081959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:43.146741 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.146635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:43.386004 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.385919 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6ww8" event={"ID":"7848077b-6e8d-41bf-a3aa-7e8ec73be770","Type":"ContainerStarted","Data":"066ae0d396a66be3e7fd670944f7a4f19b4be263ca3cf6ca9b76ea543042b6f7"} Jul 14 18:11:43.386004 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.385975 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6ww8" event={"ID":"7848077b-6e8d-41bf-a3aa-7e8ec73be770","Type":"ContainerStarted","Data":"b20ff18b8fad2616280285d444e87af78020d759bd668bb9ce44b93d6a3c300d"} Jul 14 18:11:43.407074 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.406994 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h6ww8" podStartSLOduration=65.736318784 podStartE2EDuration="1m7.406969644s" totalImagesPullingTime="1.67065086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:43.404457048 +0000 UTC m=+68.790355502" watchObservedRunningTime="2026-07-14 18:11:43.406969644 +0000 UTC m=+68.792868097" Jul 14 18:11:43.434497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.434466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:43.494149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:43.494078 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:44.312427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:44.312399 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:44.378937 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:44.378888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:44.915466 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:44.915388 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8st4g_27fe26db-4700-4c31-9f83-219402644385/serve-healthcheck-canary/0.log" Jul 14 18:11:45.395365 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:45.395329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5ksdk" event={"ID":"4aed1a3f-e696-4a12-ab89-ce7a961c4684","Type":"ContainerStarted","Data":"28eada13146c1816422b14b1dd07173c6de320b0b2d6850c39247acee85899fe"} Jul 14 18:11:45.395803 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:45.395425 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:11:45.414031 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:45.413973 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5ksdk" podStartSLOduration=66.002563324 podStartE2EDuration="1m9.413953381s" totalImagesPullingTime="3.411390057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:11:45.413558536 +0000 UTC m=+70.799456991" watchObservedRunningTime="2026-07-14 18:11:45.413953381 +0000 UTC m=+70.799851834" Jul 14 18:11:50.568600 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:50.568557 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:50.569022 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:50.568628 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:11:50.574385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:50.574358 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:50.676700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:50.676661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:50.757103 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:50.757073 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:11:55.667288 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:55.667195 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9db7c5476-q7zcs" podUID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" containerName="console" containerID="cri-o://3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f" gracePeriod=15 Jul 14 18:11:55.989638 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:55.989603 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:56.019345 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019313 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zssbx\" (UniqueName: \"kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019391 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019497 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019657 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019534 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019657 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019608 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019748 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca\") pod \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\" (UID: \"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe\") " Jul 14 18:11:56.019899 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019804 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config" (OuterVolumeSpecName: "console-config") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:56.019899 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.019859 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:56.020083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.020065 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-trusted-ca-bundle\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.020140 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.020088 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-config\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.020228 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.020199 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:56.020423 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.020404 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca" (OuterVolumeSpecName: "service-ca") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:11:56.022944 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.022893 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:56.023121 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.023099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:11:56.023407 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.023390 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx" (OuterVolumeSpecName: "kube-api-access-zssbx") pod "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" (UID: "13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe"). InnerVolumeSpecName "kube-api-access-zssbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:11:56.120992 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.120951 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.120992 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.120996 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssbx\" (UniqueName: \"kubernetes.io/projected/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-kube-api-access-zssbx\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.121193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.121013 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-oauth-serving-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.121193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.121026 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-serving-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.121193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.121039 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe-console-oauth-config\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:11:56.444858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.444727 2567 generic.go:356] "Generic (PLEG): container finished" podID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" containerID="3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f" exitCode=0 Jul 14 18:11:56.444858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.444793 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9db7c5476-q7zcs" Jul 14 18:11:56.444858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.444827 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9db7c5476-q7zcs" event={"ID":"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe","Type":"ContainerDied","Data":"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f"} Jul 14 18:11:56.445103 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.444877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9db7c5476-q7zcs" event={"ID":"13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe","Type":"ContainerDied","Data":"1adae3ebd671fe1482f914f98a94af94eb2d68ed634d7096f384f05e7af9dedc"} Jul 14 18:11:56.445103 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.444898 2567 scope.go:117] "RemoveContainer" containerID="3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f" Jul 14 18:11:56.455096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.455080 2567 scope.go:117] "RemoveContainer" containerID="3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f" Jul 14 18:11:56.455374 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:11:56.455354 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f\": container with ID starting with 3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f not found: ID does not exist" containerID="3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f" Jul 14 18:11:56.455425 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.455384 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f"} err="failed to get container status \"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f\": rpc error: code = NotFound desc = could not find container \"3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f\": container with ID starting with 3f4ce74e0a509d55139be78c449c3b97ddeca127659d287262e591856a80761f not found: ID does not exist" Jul 14 18:11:56.468684 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.468659 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:56.475539 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:56.475509 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9db7c5476-q7zcs"] Jul 14 18:11:57.930665 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:11:57.930633 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" path="/var/lib/kubelet/pods/13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe/volumes" Jul 14 18:12:02.157533 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:02.157492 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:12:02.220943 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:02.220918 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:12:02.489486 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:02.489460 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:12:10.573001 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:10.572973 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:12:10.576707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:10.576682 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bcb994478-jgh2r" Jul 14 18:12:15.235007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:15.234973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:16.402743 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:16.402711 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5ksdk" Jul 14 18:12:16.453729 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:16.453696 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:20.380558 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.380511 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74d5b69564-6whnd"] Jul 14 18:12:20.381096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.381077 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" containerName="console" Jul 14 18:12:20.381096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.381096 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" containerName="console" Jul 14 18:12:20.381228 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.381165 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d0bbd1-1783-40b5-9cfb-8e4f7ebd3ffe" containerName="console" Jul 14 18:12:20.392767 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.392740 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.395496 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.395473 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 18:12:20.395633 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.395549 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 18:12:20.395633 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.395570 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 18:12:20.396019 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.395998 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w5wzn\"" Jul 14 18:12:20.396131 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.396053 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 18:12:20.396200 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.396187 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 18:12:20.406441 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.406415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74d5b69564-6whnd"] Jul 14 18:12:20.408117 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.408095 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 18:12:20.545223 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-federate-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545280 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-metrics-client-ca\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545388 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57n99\" (UniqueName: \"kubernetes.io/projected/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-kube-api-access-57n99\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545621 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.545621 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.545478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.646869 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.646869 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-metrics-client-ca\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.646869 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57n99\" (UniqueName: \"kubernetes.io/projected/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-kube-api-access-57n99\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.646929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.647084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.647157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-federate-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.647713 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.647681 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-serving-certs-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.648072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.648039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-metrics-client-ca\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.648182 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.648132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.649712 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.649682 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.649916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.649897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-secret-telemeter-client\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.649980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.649908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-telemeter-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.649980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.649953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-federate-client-tls\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.658961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.658938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57n99\" (UniqueName: \"kubernetes.io/projected/fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6-kube-api-access-57n99\") pod \"telemeter-client-74d5b69564-6whnd\" (UID: \"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6\") " pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.707992 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.707956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" Jul 14 18:12:20.856711 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:20.856666 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74d5b69564-6whnd"] Jul 14 18:12:20.860573 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:12:20.860538 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa31bb53_e5c3_466e_bf40_e18cbbe7dcf6.slice/crio-7b0447dd8323a3b03d1283cc4f2fa8aa29fba73aac8148360b0805d44069cb9b WatchSource:0}: Error finding container 7b0447dd8323a3b03d1283cc4f2fa8aa29fba73aac8148360b0805d44069cb9b: Status 404 returned error can't find the container with id 7b0447dd8323a3b03d1283cc4f2fa8aa29fba73aac8148360b0805d44069cb9b Jul 14 18:12:21.550191 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:21.550159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" event={"ID":"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6","Type":"ContainerStarted","Data":"7b0447dd8323a3b03d1283cc4f2fa8aa29fba73aac8148360b0805d44069cb9b"} Jul 14 18:12:23.559490 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:23.559457 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" event={"ID":"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6","Type":"ContainerStarted","Data":"e09a89e79cdf74e1f513849bdb06db924617053b4895d3ba9cea74d93a1bddef"} Jul 14 18:12:24.565655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:24.565612 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" event={"ID":"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6","Type":"ContainerStarted","Data":"b25b7e09529e3c4795e386ce9d49aa2ced00b163f97e09f7a08626a626c20d8b"} Jul 14 18:12:24.566052 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:24.565681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" event={"ID":"fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6","Type":"ContainerStarted","Data":"887c47023460a6ad15e28f6e32b044bd298dbd212dc44e35ce37fd8c8fb03011"} Jul 14 18:12:24.592309 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:24.592202 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74d5b69564-6whnd" podStartSLOduration=1.971883026 podStartE2EDuration="4.592186693s" totalImagesPullingTime="2.620303667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:24.591515158 +0000 UTC m=+109.977413601" watchObservedRunningTime="2026-07-14 18:12:24.592186693 +0000 UTC m=+109.978085145" Jul 14 18:12:25.484932 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:25.484893 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:25.557856 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:25.557792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:30.764484 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:30.764434 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:30.868076 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:30.868043 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:33.156821 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:33.156775 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:33.218870 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:33.218832 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:35.974698 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:35.974664 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:12:36.092933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:36.092839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:36.193200 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:36.193163 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:36.430119 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:36.430040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:36.493719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:36.493685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:42.661425 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:42.661389 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:42.719381 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:42.719347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:47.209699 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:47.209665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:47.310378 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:47.310347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:58.092764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:58.092730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:58.194975 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:58.194934 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:58.279958 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:58.279924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:59.921586 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:59.921554 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:12:59.980438 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:12:59.980406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:13:00.995677 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:00.995634 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4b4644fd-mcwjq" podUID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" containerName="console" containerID="cri-o://dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13" gracePeriod=15 Jul 14 18:13:01.249696 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.249644 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:13:01.307770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.307733 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.307770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.307772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308023 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.307793 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308023 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.307844 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308136 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308033 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308136 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308109 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308241 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308159 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2mmm\" (UniqueName: \"kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm\") pod \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\" (UID: \"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8\") " Jul 14 18:13:01.308241 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308189 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config" (OuterVolumeSpecName: "console-config") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:01.308241 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308216 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca" (OuterVolumeSpecName: "service-ca") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:01.308241 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308226 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:01.308421 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308265 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:13:01.308498 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308479 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-config\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.308563 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308505 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-oauth-serving-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.308563 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308517 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-trusted-ca-bundle\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.308563 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.308526 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.310186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.310150 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:13:01.310186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.310157 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm" (OuterVolumeSpecName: "kube-api-access-r2mmm") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "kube-api-access-r2mmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:13:01.310335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.310199 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" (UID: "561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:13:01.409427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.409385 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-oauth-config\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.409427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.409416 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-console-serving-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.409427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.409425 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2mmm\" (UniqueName: \"kubernetes.io/projected/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8-kube-api-access-r2mmm\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:13:01.721620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.721527 2567 generic.go:356] "Generic (PLEG): container finished" podID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" containerID="dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13" exitCode=0 Jul 14 18:13:01.721620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.721568 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b4644fd-mcwjq" event={"ID":"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8","Type":"ContainerDied","Data":"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13"} Jul 14 18:13:01.721620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.721574 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4b4644fd-mcwjq" Jul 14 18:13:01.721620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.721617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4b4644fd-mcwjq" event={"ID":"561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8","Type":"ContainerDied","Data":"c7bfdc96c8716a7a18fa68a01d3911e43254ca782c125d9d8cec878bd7d58595"} Jul 14 18:13:01.721952 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.721641 2567 scope.go:117] "RemoveContainer" containerID="dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13" Jul 14 18:13:01.730868 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.730848 2567 scope.go:117] "RemoveContainer" containerID="dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13" Jul 14 18:13:01.731123 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:13:01.731104 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13\": container with ID starting with dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13 not found: ID does not exist" containerID="dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13" Jul 14 18:13:01.731173 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.731132 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13"} err="failed to get container status \"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13\": rpc error: code = NotFound desc = could not find container \"dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13\": container with ID starting with dae8810db053a8184d2cb6b7a514d8a8b3d93b74119956fc96acce5f453c3a13 not found: ID does not exist" Jul 14 18:13:01.746214 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.746182 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:13:01.751098 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.751076 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4b4644fd-mcwjq"] Jul 14 18:13:01.930040 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:01.930006 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" path="/var/lib/kubelet/pods/561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8/volumes" Jul 14 18:13:15.186042 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:15.186002 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:13:15.254585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:15.254549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:13:58.460122 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:58.460087 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:13:58.557838 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:13:58.557779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:20.516217 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:20.516181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:21.265867 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:21.265826 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:21.391784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:21.391752 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:24.312516 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:24.312469 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:24.424688 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:24.424655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:25.713126 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:25.713097 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:25.807455 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:25.807420 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:26.212146 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:26.212114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:26.310787 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:26.310748 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:27.309222 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.309188 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2qsf"] Jul 14 18:14:27.309585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.309523 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" containerName="console" Jul 14 18:14:27.309585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.309533 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" containerName="console" Jul 14 18:14:27.309653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.309597 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="561a5fe8-0f84-4350-98eb-f0b9d2ebaaa8" containerName="console" Jul 14 18:14:27.311606 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.311585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.314316 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.314296 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 18:14:27.314481 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.314464 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 18:14:27.315075 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.315060 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2x2zj\"" Jul 14 18:14:27.323225 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.323193 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2qsf"] Jul 14 18:14:27.363717 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.363667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mld7r\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-kube-api-access-mld7r\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.363928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.363797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.464905 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.464867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.465070 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.464933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mld7r\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-kube-api-access-mld7r\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.474840 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.474795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.474977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.474901 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mld7r\" (UniqueName: \"kubernetes.io/projected/9a90ebc3-af07-479b-b8e6-e74896823836-kube-api-access-mld7r\") pod \"cert-manager-webhook-6486ffd997-f2qsf\" (UID: \"9a90ebc3-af07-479b-b8e6-e74896823836\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.634366 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.634273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:27.775332 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:27.775305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f2qsf"] Jul 14 18:14:27.777666 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:14:27.777640 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a90ebc3_af07_479b_b8e6_e74896823836.slice/crio-78f640523b58940e012ad6f591afdcf0df11662f9f099b6fd6bb028b285ddb4f WatchSource:0}: Error finding container 78f640523b58940e012ad6f591afdcf0df11662f9f099b6fd6bb028b285ddb4f: Status 404 returned error can't find the container with id 78f640523b58940e012ad6f591afdcf0df11662f9f099b6fd6bb028b285ddb4f Jul 14 18:14:28.056725 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:28.056681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" event={"ID":"9a90ebc3-af07-479b-b8e6-e74896823836","Type":"ContainerStarted","Data":"78f640523b58940e012ad6f591afdcf0df11662f9f099b6fd6bb028b285ddb4f"} Jul 14 18:14:28.446205 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:28.446096 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:28.858668 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:28.858630 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:30.904784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:30.904683 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:31.005149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:31.005113 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:31.071354 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:31.071321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" event={"ID":"9a90ebc3-af07-479b-b8e6-e74896823836","Type":"ContainerStarted","Data":"091f2340de2e5e394b2f0d5791853fa0beb46694bc811632d1d8a4fa44a633a0"} Jul 14 18:14:31.071558 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:31.071395 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:31.093138 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:31.093090 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" podStartSLOduration=1.342093575 podStartE2EDuration="4.093074852s" totalImagesPullingTime="2.750981277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:31.092395642 +0000 UTC m=+236.478294095" watchObservedRunningTime="2026-07-14 18:14:31.093074852 +0000 UTC m=+236.478973305" Jul 14 18:14:36.362590 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.362517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:36.456576 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.456542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:36.515617 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.515583 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:36.653131 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.653049 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rb45q"] Jul 14 18:14:36.655758 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.655742 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.657931 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.657914 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-44bkw\"" Jul 14 18:14:36.666314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.666289 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rb45q"] Jul 14 18:14:36.763637 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.763601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vf2\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-kube-api-access-57vf2\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.763637 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.763644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.864783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.864751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vf2\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-kube-api-access-57vf2\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.864783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.864786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.874839 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.874787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.874978 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.874866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vf2\" (UniqueName: \"kubernetes.io/projected/6612e47a-45d1-4a1a-b66a-a9653c9586fd-kube-api-access-57vf2\") pod \"cert-manager-55b4d954d5-rb45q\" (UID: \"6612e47a-45d1-4a1a-b66a-a9653c9586fd\") " pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:36.964423 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:36.964333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rb45q" Jul 14 18:14:37.076876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:37.076847 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f2qsf" Jul 14 18:14:37.098291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:37.098260 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rb45q"] Jul 14 18:14:37.099933 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:14:37.099906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6612e47a_45d1_4a1a_b66a_a9653c9586fd.slice/crio-61e01cf867f73bcb2ae3d6abe26394424f10fc4cde832f1d24f1ac0b38c94f97 WatchSource:0}: Error finding container 61e01cf867f73bcb2ae3d6abe26394424f10fc4cde832f1d24f1ac0b38c94f97: Status 404 returned error can't find the container with id 61e01cf867f73bcb2ae3d6abe26394424f10fc4cde832f1d24f1ac0b38c94f97 Jul 14 18:14:38.106117 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:38.106074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rb45q" event={"ID":"6612e47a-45d1-4a1a-b66a-a9653c9586fd","Type":"ContainerStarted","Data":"c222c6123f48f9510ddf3ac4a7ce74771cae9c7deac697084f63befce5582e76"} Jul 14 18:14:38.106488 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:38.106126 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rb45q" event={"ID":"6612e47a-45d1-4a1a-b66a-a9653c9586fd","Type":"ContainerStarted","Data":"61e01cf867f73bcb2ae3d6abe26394424f10fc4cde832f1d24f1ac0b38c94f97"} Jul 14 18:14:38.125665 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:38.125609 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rb45q" podStartSLOduration=2.125591463 podStartE2EDuration="2.125591463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:38.124991657 +0000 UTC m=+243.510890112" watchObservedRunningTime="2026-07-14 18:14:38.125591463 +0000 UTC m=+243.511489913" Jul 14 18:14:39.307076 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:39.307037 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:39.412195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:39.412158 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:40.787103 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.787068 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs"] Jul 14 18:14:40.789938 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.789915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:40.792474 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.792447 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:14:40.792598 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.792533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 18:14:40.793190 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.793175 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hm27d\"" Jul 14 18:14:40.803723 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.803701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs"] Jul 14 18:14:40.898614 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.898582 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:40.898784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.898657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fcg\" (UniqueName: \"kubernetes.io/projected/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-kube-api-access-s9fcg\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:40.999609 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.999579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fcg\" (UniqueName: \"kubernetes.io/projected/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-kube-api-access-s9fcg\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:40.999759 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:40.999635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:41.000053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:41.000035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:41.009618 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:41.009597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fcg\" (UniqueName: \"kubernetes.io/projected/7bb08805-5a28-4db4-b5f4-cfd66a1ee67d-kube-api-access-s9fcg\") pod \"openshift-lws-operator-fd8ccff4c-t8bxs\" (UID: \"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:41.099621 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:41.099521 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" Jul 14 18:14:41.225455 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:41.225426 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs"] Jul 14 18:14:41.228078 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:14:41.228048 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bb08805_5a28_4db4_b5f4_cfd66a1ee67d.slice/crio-dd2c413b7f8ca8e8071a589e4c6ea312ebdd3314e0a77a89c82aa6c30089ed84 WatchSource:0}: Error finding container dd2c413b7f8ca8e8071a589e4c6ea312ebdd3314e0a77a89c82aa6c30089ed84: Status 404 returned error can't find the container with id dd2c413b7f8ca8e8071a589e4c6ea312ebdd3314e0a77a89c82aa6c30089ed84 Jul 14 18:14:42.131262 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:42.131223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" event={"ID":"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d","Type":"ContainerStarted","Data":"dd2c413b7f8ca8e8071a589e4c6ea312ebdd3314e0a77a89c82aa6c30089ed84"} Jul 14 18:14:44.141454 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:44.141425 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" event={"ID":"7bb08805-5a28-4db4-b5f4-cfd66a1ee67d","Type":"ContainerStarted","Data":"d632ba4f57b51550dbc2237761fac5c00ea24f7dc29f1f83b408530b0506cc94"} Jul 14 18:14:44.166378 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:44.166317 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t8bxs" podStartSLOduration=1.859830294 podStartE2EDuration="4.166294928s" totalImagesPullingTime="2.306464634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:44.165464386 +0000 UTC m=+249.551362832" watchObservedRunningTime="2026-07-14 18:14:44.166294928 +0000 UTC m=+249.552193383" Jul 14 18:14:44.965731 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:44.965661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:45.058580 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:45.058549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:14:46.757106 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:14:46.757036 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:01.750880 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:01.750779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:03.772068 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:03.772038 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:03.901094 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:03.901063 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:04.191210 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:04.191136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:04.845677 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:04.845646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:04.951539 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:04.951510 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:05.121835 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.121739 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt"] Jul 14 18:15:05.126995 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.126975 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.134772 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.134747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:15:05.135845 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.135825 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-52hn2\"" Jul 14 18:15:05.135985 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.135968 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:15:05.136413 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.136400 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:15:05.136471 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.136462 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:15:05.178559 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.178523 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt"] Jul 14 18:15:05.212096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.212061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.212265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.212110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.212265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.212133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.212265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.212245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6msd\" (UniqueName: \"kubernetes.io/projected/4ef798df-dc29-4dc3-b23a-fc2274e34761-kube-api-access-t6msd\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.212403 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.212314 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313297 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313297 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313565 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313340 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313565 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313565 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6msd\" (UniqueName: \"kubernetes.io/projected/4ef798df-dc29-4dc3-b23a-fc2274e34761-kube-api-access-t6msd\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.313800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.313775 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-manifests\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.315730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.315705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ef798df-dc29-4dc3-b23a-fc2274e34761-tmp\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.315957 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.315940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-webhook-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.316030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.315977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ef798df-dc29-4dc3-b23a-fc2274e34761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.324474 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.324446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6msd\" (UniqueName: \"kubernetes.io/projected/4ef798df-dc29-4dc3-b23a-fc2274e34761-kube-api-access-t6msd\") pod \"opendatahub-operator-controller-manager-557b7f646c-86prt\" (UID: \"4ef798df-dc29-4dc3-b23a-fc2274e34761\") " pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.385844 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.385729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:05.440926 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.440896 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:05.478405 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.478279 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:05.584652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.584625 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt"] Jul 14 18:15:05.586851 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:15:05.586824 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef798df_dc29_4dc3_b23a_fc2274e34761.slice/crio-9e32b968c640f7f6db8f2a082d38bbb00ef5cadf2ba27c2a8bbcf1abf1c44dc5 WatchSource:0}: Error finding container 9e32b968c640f7f6db8f2a082d38bbb00ef5cadf2ba27c2a8bbcf1abf1c44dc5: Status 404 returned error can't find the container with id 9e32b968c640f7f6db8f2a082d38bbb00ef5cadf2ba27c2a8bbcf1abf1c44dc5 Jul 14 18:15:05.695972 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.695887 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:05.792511 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:05.792479 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:06.233131 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:06.233089 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" event={"ID":"4ef798df-dc29-4dc3-b23a-fc2274e34761","Type":"ContainerStarted","Data":"9e32b968c640f7f6db8f2a082d38bbb00ef5cadf2ba27c2a8bbcf1abf1c44dc5"} Jul 14 18:15:09.249134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:09.249096 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" event={"ID":"4ef798df-dc29-4dc3-b23a-fc2274e34761","Type":"ContainerStarted","Data":"348e10e5d7ccfac22c1f686c16caf6a880698cba773cceaa8f0394fb30e860ab"} Jul 14 18:15:10.254749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:10.254713 2567 generic.go:356] "Generic (PLEG): container finished" podID="4ef798df-dc29-4dc3-b23a-fc2274e34761" containerID="348e10e5d7ccfac22c1f686c16caf6a880698cba773cceaa8f0394fb30e860ab" exitCode=0 Jul 14 18:15:10.255161 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:10.254796 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" event={"ID":"4ef798df-dc29-4dc3-b23a-fc2274e34761","Type":"ContainerDied","Data":"348e10e5d7ccfac22c1f686c16caf6a880698cba773cceaa8f0394fb30e860ab"} Jul 14 18:15:11.046418 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:11.046362 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:11.143612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:11.143576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:11.261578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:11.261542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" event={"ID":"4ef798df-dc29-4dc3-b23a-fc2274e34761","Type":"ContainerStarted","Data":"c65e8ce31ce8b91b840dfdffe28ca4638b36ac530d9b8bb559e4aef6884327cc"} Jul 14 18:15:11.262075 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:11.261673 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:16.762099 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:16.762053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:16.861806 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:16.861776 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:22.268094 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:22.268065 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" Jul 14 18:15:22.294456 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:22.294400 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-557b7f646c-86prt" podStartSLOduration=12.830805059 podStartE2EDuration="17.294381261s" totalImagesPullingTime="3.45760129s" totalInitContainerRuntime="1.005974912s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:11.314443632 +0000 UTC m=+276.700342086" watchObservedRunningTime="2026-07-14 18:15:22.294381261 +0000 UTC m=+287.680279713" Jul 14 18:15:24.455837 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:24.455782 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:24.574732 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:24.574694 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:25.338631 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:25.338599 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:25.436973 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:25.436935 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:25.536491 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:25.536443 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:25.637714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:25.637596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:25.737017 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:25.736978 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:26.091169 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:26.091125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:26.263731 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:26.263698 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:33.778550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.778509 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m"] Jul 14 18:15:33.781561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.781542 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.784063 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.784037 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 18:15:33.784239 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.784222 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 18:15:33.784770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.784752 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-strzs\"" Jul 14 18:15:33.784921 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.784783 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 18:15:33.792939 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.792912 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m"] Jul 14 18:15:33.880409 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.880338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfrg\" (UniqueName: \"kubernetes.io/projected/ade5c338-5d02-4aec-be76-f71fb89e6fcc-kube-api-access-8kfrg\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.880581 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.880423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.880581 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.880466 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ade5c338-5d02-4aec-be76-f71fb89e6fcc-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.880581 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.880495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.982036 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.981993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.982203 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.982091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfrg\" (UniqueName: \"kubernetes.io/projected/ade5c338-5d02-4aec-be76-f71fb89e6fcc-kube-api-access-8kfrg\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.982203 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.982147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.982285 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.982210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ade5c338-5d02-4aec-be76-f71fb89e6fcc-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.982967 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.982940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ade5c338-5d02-4aec-be76-f71fb89e6fcc-manager-config\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.984504 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.984482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-metrics-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.984581 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.984504 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ade5c338-5d02-4aec-be76-f71fb89e6fcc-cert\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:33.994723 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:33.994693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfrg\" (UniqueName: \"kubernetes.io/projected/ade5c338-5d02-4aec-be76-f71fb89e6fcc-kube-api-access-8kfrg\") pod \"lws-controller-manager-55b7dd5b8c-5w45m\" (UID: \"ade5c338-5d02-4aec-be76-f71fb89e6fcc\") " pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:34.091044 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:34.090953 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:34.225825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:34.225779 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m"] Jul 14 18:15:34.228226 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:15:34.228194 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade5c338_5d02_4aec_be76_f71fb89e6fcc.slice/crio-e152b88f594f461b1a3f04ff95844297f5d981d1fcafb00ce018db7ad9a53060 WatchSource:0}: Error finding container e152b88f594f461b1a3f04ff95844297f5d981d1fcafb00ce018db7ad9a53060: Status 404 returned error can't find the container with id e152b88f594f461b1a3f04ff95844297f5d981d1fcafb00ce018db7ad9a53060 Jul 14 18:15:34.365208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:34.365122 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" event={"ID":"ade5c338-5d02-4aec-be76-f71fb89e6fcc","Type":"ContainerStarted","Data":"e152b88f594f461b1a3f04ff95844297f5d981d1fcafb00ce018db7ad9a53060"} Jul 14 18:15:35.441451 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.441413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:35.539234 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.539199 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:35.592198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.592024 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-fwm48"] Jul 14 18:15:35.595459 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.595430 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.602491 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.602435 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 18:15:35.603165 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.603128 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:15:35.603376 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.603344 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 18:15:35.603719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.603695 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:15:35.603852 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.603795 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mxn2c\"" Jul 14 18:15:35.610669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.610599 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-fwm48"] Jul 14 18:15:35.633321 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.633251 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:35.695995 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.695904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4c8035-e58d-45d4-b82e-513983279c85-tls-certs\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.696162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.696088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjb8\" (UniqueName: \"kubernetes.io/projected/2f4c8035-e58d-45d4-b82e-513983279c85-kube-api-access-4qjb8\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.696162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.696127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4c8035-e58d-45d4-b82e-513983279c85-tmp\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.730149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.730105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:35.797531 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.797497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjb8\" (UniqueName: \"kubernetes.io/projected/2f4c8035-e58d-45d4-b82e-513983279c85-kube-api-access-4qjb8\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.797703 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.797547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4c8035-e58d-45d4-b82e-513983279c85-tmp\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.797703 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.797601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4c8035-e58d-45d4-b82e-513983279c85-tls-certs\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.800084 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.800054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4c8035-e58d-45d4-b82e-513983279c85-tmp\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.813242 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.813212 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 18:15:35.820893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.820865 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:15:35.821567 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.821515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4c8035-e58d-45d4-b82e-513983279c85-tls-certs\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.831034 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.831008 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:15:35.840645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.840616 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:35.842033 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.842004 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjb8\" (UniqueName: \"kubernetes.io/projected/2f4c8035-e58d-45d4-b82e-513983279c85-kube-api-access-4qjb8\") pod \"kube-auth-proxy-565f78b694-fwm48\" (UID: \"2f4c8035-e58d-45d4-b82e-513983279c85\") " pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.910300 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.910154 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:15:35.924079 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.910841 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:15:35.924079 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.911731 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mxn2c\"" Jul 14 18:15:35.924079 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.916638 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:15:35.924079 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.918828 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" Jul 14 18:15:35.931714 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:35.931682 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:36.074370 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:36.074337 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565f78b694-fwm48"] Jul 14 18:15:36.078369 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:15:36.077631 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4c8035_e58d_45d4_b82e_513983279c85.slice/crio-4d044e7e9ed2f53a82fa764ddf2d89c7e0e92b700c14f42298f9c0a9d8076b99 WatchSource:0}: Error finding container 4d044e7e9ed2f53a82fa764ddf2d89c7e0e92b700c14f42298f9c0a9d8076b99: Status 404 returned error can't find the container with id 4d044e7e9ed2f53a82fa764ddf2d89c7e0e92b700c14f42298f9c0a9d8076b99 Jul 14 18:15:36.385391 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:36.385341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" event={"ID":"ade5c338-5d02-4aec-be76-f71fb89e6fcc","Type":"ContainerStarted","Data":"a097f4c4d0d12a14f6350092320380495b6697b8e0ddbdbed3cb67cb1ae86c78"} Jul 14 18:15:36.385594 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:36.385544 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:15:36.386503 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:36.386476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" event={"ID":"2f4c8035-e58d-45d4-b82e-513983279c85","Type":"ContainerStarted","Data":"4d044e7e9ed2f53a82fa764ddf2d89c7e0e92b700c14f42298f9c0a9d8076b99"} Jul 14 18:15:36.415377 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:36.415309 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" podStartSLOduration=1.709024165 podStartE2EDuration="3.415291831s" totalImagesPullingTime="1.706267666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:36.414118592 +0000 UTC m=+301.800017046" watchObservedRunningTime="2026-07-14 18:15:36.415291831 +0000 UTC m=+301.801190283" Jul 14 18:15:37.444008 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:37.443974 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:37.513832 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:37.513780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:38.876017 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:38.875975 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:40.831189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:40.831168 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 18:15:41.413237 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:41.413198 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" event={"ID":"2f4c8035-e58d-45d4-b82e-513983279c85","Type":"ContainerStarted","Data":"a77eb66fa35606f42f0c163e0db94465502e3a8e082a5832b34844b3c5bcd09d"} Jul 14 18:15:41.432153 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:41.432102 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-565f78b694-fwm48" podStartSLOduration=1.683867776 podStartE2EDuration="6.43208754s" totalImagesPullingTime="4.748219764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:15:41.430326786 +0000 UTC m=+306.816225240" watchObservedRunningTime="2026-07-14 18:15:41.43208754 +0000 UTC m=+306.817985992" Jul 14 18:15:41.536825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:41.536774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:41.637644 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:41.637585 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:42.561524 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:42.561491 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:42.870314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:42.870231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:43.148005 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:43.147924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:15:47.394272 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:15:47.394240 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55b7dd5b8c-5w45m" Jul 14 18:16:26.071885 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:26.071845 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:26.146189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:26.146159 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:26.204152 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:26.204121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:40.555080 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:40.555044 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:40.624083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:40.624052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:42.395883 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:42.395839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:55.528700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:55.528661 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:58.316448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:58.316408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:16:59.718271 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:16:59.718236 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:02.160252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:02.160219 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:02.388913 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:02.388877 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:02.717979 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:02.717951 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:06.138253 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:06.138217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:07.182824 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:07.182792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:09.182085 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:09.182051 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:10.979668 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:10.979634 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:12.335455 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:12.335425 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:12.439948 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:12.439920 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:16.207230 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:16.207194 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:16.305710 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:16.305678 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:20.321146 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:20.321109 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:21.885104 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:21.885067 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:23.763416 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:23.763387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:23.863092 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:23.863031 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:30.396709 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.396674 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:30.489232 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.489203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:30.692470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.692379 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg"] Jul 14 18:17:30.695358 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.695334 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.697450 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.697426 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:17:30.697450 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.697426 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:17:30.697623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.697489 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k5cr2\"" Jul 14 18:17:30.709858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.709833 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg"] Jul 14 18:17:30.758029 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.757996 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dgx\" (UniqueName: \"kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.758200 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.758034 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.795515 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.795483 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg"] Jul 14 18:17:30.795782 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:17:30.795763 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-w4dgx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" podUID="e2961aeb-8264-49c1-816f-c0334bff8dac" Jul 14 18:17:30.859196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.859161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dgx\" (UniqueName: \"kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.859196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.859200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.859600 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.859584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.868795 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.868766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dgx\" (UniqueName: \"kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.913262 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.913232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:30.919907 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:30.919886 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:31.061141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.061112 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4dgx\" (UniqueName: \"kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx\") pod \"e2961aeb-8264-49c1-816f-c0334bff8dac\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " Jul 14 18:17:31.061312 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.061158 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume\") pod \"e2961aeb-8264-49c1-816f-c0334bff8dac\" (UID: \"e2961aeb-8264-49c1-816f-c0334bff8dac\") " Jul 14 18:17:31.061494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.061469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e2961aeb-8264-49c1-816f-c0334bff8dac" (UID: "e2961aeb-8264-49c1-816f-c0334bff8dac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:17:31.063269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.063243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx" (OuterVolumeSpecName: "kube-api-access-w4dgx") pod "e2961aeb-8264-49c1-816f-c0334bff8dac" (UID: "e2961aeb-8264-49c1-816f-c0334bff8dac"). InnerVolumeSpecName "kube-api-access-w4dgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:31.161909 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.161870 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2961aeb-8264-49c1-816f-c0334bff8dac-extensions-socket-volume\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.161909 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.161902 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dgx\" (UniqueName: \"kubernetes.io/projected/e2961aeb-8264-49c1-816f-c0334bff8dac-kube-api-access-w4dgx\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:17:31.317849 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.317754 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg"] Jul 14 18:17:31.324445 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.324414 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg"] Jul 14 18:17:31.365515 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.365477 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:31.368764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.368737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.415423 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.415366 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:31.464556 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.464514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.464702 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.464635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprbr\" (UniqueName: \"kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.565114 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.565075 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.565283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.565185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprbr\" (UniqueName: \"kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.565474 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.565453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.575642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.575581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprbr\" (UniqueName: \"kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jmjr6\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.680036 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.680000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:31.823628 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.823500 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:31.826698 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:17:31.826638 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996a6cf8_7a84_4921_bbf7_17e32bbb86b4.slice/crio-396ef730bc911f8dcd665202b0237b877b22a530b6bb464f9859b255b1fb3a42 WatchSource:0}: Error finding container 396ef730bc911f8dcd665202b0237b877b22a530b6bb464f9859b255b1fb3a42: Status 404 returned error can't find the container with id 396ef730bc911f8dcd665202b0237b877b22a530b6bb464f9859b255b1fb3a42 Jul 14 18:17:31.828962 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.828949 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:17:31.918713 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.918681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" event={"ID":"996a6cf8-7a84-4921-bbf7-17e32bbb86b4","Type":"ContainerStarted","Data":"396ef730bc911f8dcd665202b0237b877b22a530b6bb464f9859b255b1fb3a42"} Jul 14 18:17:31.918713 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.918708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:31.920836 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:17:31.920793 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="e2961aeb-8264-49c1-816f-c0334bff8dac" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:31.927120 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:17:31.927090 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-b5jxg\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="e2961aeb-8264-49c1-816f-c0334bff8dac" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b5jxg" Jul 14 18:17:31.930262 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:31.930240 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2961aeb-8264-49c1-816f-c0334bff8dac" path="/var/lib/kubelet/pods/e2961aeb-8264-49c1-816f-c0334bff8dac/volumes" Jul 14 18:17:33.229546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:33.229487 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:33.335912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:33.335882 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:37.950238 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:37.950146 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" event={"ID":"996a6cf8-7a84-4921-bbf7-17e32bbb86b4","Type":"ContainerStarted","Data":"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5"} Jul 14 18:17:37.950625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:37.950259 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:37.991464 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:37.991407 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" podStartSLOduration=1.313014696 podStartE2EDuration="6.991389271s" totalImagesPullingTime="5.678374575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:17:37.99084551 +0000 UTC m=+423.376743962" watchObservedRunningTime="2026-07-14 18:17:37.991389271 +0000 UTC m=+423.377287723" Jul 14 18:17:38.023141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:38.023111 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:38.121334 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:38.121301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:41.528029 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:41.527990 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:41.628961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:41.628924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:43.694860 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.694823 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:43.695284 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.695003 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" podUID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" containerName="manager" containerID="cri-o://851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5" gracePeriod=10 Jul 14 18:17:43.696719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.696689 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:43.939778 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.939753 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:43.978354 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.978321 2567 generic.go:356] "Generic (PLEG): container finished" podID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" containerID="851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5" exitCode=0 Jul 14 18:17:43.978531 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.978370 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" Jul 14 18:17:43.978531 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.978370 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" event={"ID":"996a6cf8-7a84-4921-bbf7-17e32bbb86b4","Type":"ContainerDied","Data":"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5"} Jul 14 18:17:43.978531 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.978436 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6" event={"ID":"996a6cf8-7a84-4921-bbf7-17e32bbb86b4","Type":"ContainerDied","Data":"396ef730bc911f8dcd665202b0237b877b22a530b6bb464f9859b255b1fb3a42"} Jul 14 18:17:43.978531 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.978468 2567 scope.go:117] "RemoveContainer" containerID="851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5" Jul 14 18:17:43.980771 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.980755 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mprbr\" (UniqueName: \"kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr\") pod \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " Jul 14 18:17:43.980961 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.980936 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume\") pod \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\" (UID: \"996a6cf8-7a84-4921-bbf7-17e32bbb86b4\") " Jul 14 18:17:43.981522 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.981493 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "996a6cf8-7a84-4921-bbf7-17e32bbb86b4" (UID: "996a6cf8-7a84-4921-bbf7-17e32bbb86b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:17:43.983168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.983145 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr" (OuterVolumeSpecName: "kube-api-access-mprbr") pod "996a6cf8-7a84-4921-bbf7-17e32bbb86b4" (UID: "996a6cf8-7a84-4921-bbf7-17e32bbb86b4"). InnerVolumeSpecName "kube-api-access-mprbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:17:43.988733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.988713 2567 scope.go:117] "RemoveContainer" containerID="851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5" Jul 14 18:17:43.988998 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:17:43.988976 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5\": container with ID starting with 851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5 not found: ID does not exist" containerID="851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5" Jul 14 18:17:43.989063 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:43.989002 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5"} err="failed to get container status \"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5\": rpc error: code = NotFound desc = could not find container \"851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5\": container with ID starting with 851e6261a404568a7588a5279fa87140bd0e12cf9200ab58d96f215abb8e26e5 not found: ID does not exist" Jul 14 18:17:44.082289 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:44.082253 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-extensions-socket-volume\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:17:44.082289 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:44.082283 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprbr\" (UniqueName: \"kubernetes.io/projected/996a6cf8-7a84-4921-bbf7-17e32bbb86b4-kube-api-access-mprbr\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:17:44.301158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:44.301127 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:44.310569 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:44.310543 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jmjr6"] Jul 14 18:17:45.930579 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:45.930544 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" path="/var/lib/kubelet/pods/996a6cf8-7a84-4921-bbf7-17e32bbb86b4/volumes" Jul 14 18:17:48.206549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:48.206517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:48.309592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:48.309560 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:51.391778 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:51.391743 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:51.489829 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:51.489792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:56.001262 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:56.001225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:56.102708 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:56.102675 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:56.210255 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:56.210216 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:56.298622 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:56.298542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:17:56.421082 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:17:56.421011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:00.128885 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:00.128841 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:00.322793 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:00.322762 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:01.269040 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:01.269004 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:01.369477 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:01.369442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:05.105665 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:05.105631 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:05.178087 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:05.178059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:13.392419 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.392387 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:13.411501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.411473 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:13.411870 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.411857 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" containerName="manager" Jul 14 18:18:13.411919 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.411871 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" containerName="manager" Jul 14 18:18:13.411952 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.411940 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="996a6cf8-7a84-4921-bbf7-17e32bbb86b4" containerName="manager" Jul 14 18:18:13.416010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.415995 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.418249 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.418229 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n85pf\"" Jul 14 18:18:13.418365 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.418246 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:18:13.418365 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.418272 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 18:18:13.418742 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.418718 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:18:13.429987 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.429961 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:13.501599 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.501564 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:13.502898 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.502873 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:13.548532 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.548496 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwbf\" (UniqueName: \"kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.548715 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.548541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.599986 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.599959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:13.649321 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.649233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwbf\" (UniqueName: \"kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.649321 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.649277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.649914 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.649896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.658775 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.658759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwbf\" (UniqueName: \"kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf\") pod \"limitador-limitador-64c94fd64f-glzrl\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.726526 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.726489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:13.868567 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:13.868534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:13.872155 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:18:13.872130 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded974897_0ac7_4891_be61_010048135a3b.slice/crio-c512d299f883e9324801e6dc5926585544e88a93f4f2af009b35bfbc397d8dbd WatchSource:0}: Error finding container c512d299f883e9324801e6dc5926585544e88a93f4f2af009b35bfbc397d8dbd: Status 404 returned error can't find the container with id c512d299f883e9324801e6dc5926585544e88a93f4f2af009b35bfbc397d8dbd Jul 14 18:18:14.124689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:14.124656 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" event={"ID":"ed974897-0ac7-4891-be61-010048135a3b","Type":"ContainerStarted","Data":"c512d299f883e9324801e6dc5926585544e88a93f4f2af009b35bfbc397d8dbd"} Jul 14 18:18:14.430767 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:14.430689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:14.523311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:14.523225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:17.143715 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:17.143682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" event={"ID":"ed974897-0ac7-4891-be61-010048135a3b","Type":"ContainerStarted","Data":"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce"} Jul 14 18:18:17.144199 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:17.143780 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:17.165186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:17.165134 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" podStartSLOduration=1.5891017509999998 podStartE2EDuration="4.165118399s" totalImagesPullingTime="2.576016648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:17.164172325 +0000 UTC m=+462.550070779" watchObservedRunningTime="2026-07-14 18:18:17.165118399 +0000 UTC m=+462.551016851" Jul 14 18:18:28.149535 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:28.149496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:28.883590 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:28.883554 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:28.883800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:28.883776 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" podUID="ed974897-0ac7-4891-be61-010048135a3b" containerName="limitador" containerID="cri-o://51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce" gracePeriod=30 Jul 14 18:18:29.426480 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.426455 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:29.488227 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.488188 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file\") pod \"ed974897-0ac7-4891-be61-010048135a3b\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " Jul 14 18:18:29.488396 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.488272 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwbf\" (UniqueName: \"kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf\") pod \"ed974897-0ac7-4891-be61-010048135a3b\" (UID: \"ed974897-0ac7-4891-be61-010048135a3b\") " Jul 14 18:18:29.488549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.488524 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file" (OuterVolumeSpecName: "config-file") pod "ed974897-0ac7-4891-be61-010048135a3b" (UID: "ed974897-0ac7-4891-be61-010048135a3b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:18:29.490293 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.490264 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf" (OuterVolumeSpecName: "kube-api-access-ltwbf") pod "ed974897-0ac7-4891-be61-010048135a3b" (UID: "ed974897-0ac7-4891-be61-010048135a3b"). InnerVolumeSpecName "kube-api-access-ltwbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:18:29.589510 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.589459 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed974897-0ac7-4891-be61-010048135a3b-config-file\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:18:29.589510 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:29.589504 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwbf\" (UniqueName: \"kubernetes.io/projected/ed974897-0ac7-4891-be61-010048135a3b-kube-api-access-ltwbf\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:18:30.203189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.203098 2567 generic.go:356] "Generic (PLEG): container finished" podID="ed974897-0ac7-4891-be61-010048135a3b" containerID="51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce" exitCode=0 Jul 14 18:18:30.203189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.203146 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" Jul 14 18:18:30.203189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.203155 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" event={"ID":"ed974897-0ac7-4891-be61-010048135a3b","Type":"ContainerDied","Data":"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce"} Jul 14 18:18:30.203422 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.203207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-glzrl" event={"ID":"ed974897-0ac7-4891-be61-010048135a3b","Type":"ContainerDied","Data":"c512d299f883e9324801e6dc5926585544e88a93f4f2af009b35bfbc397d8dbd"} Jul 14 18:18:30.203422 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.203240 2567 scope.go:117] "RemoveContainer" containerID="51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce" Jul 14 18:18:30.211466 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.211440 2567 scope.go:117] "RemoveContainer" containerID="51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce" Jul 14 18:18:30.211708 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:18:30.211688 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce\": container with ID starting with 51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce not found: ID does not exist" containerID="51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce" Jul 14 18:18:30.211757 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.211718 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce"} err="failed to get container status \"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce\": rpc error: code = NotFound desc = could not find container \"51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce\": container with ID starting with 51a00a8260b17e9f3bb65aae6e11c9b611279bec7dde94e6b114cd0c875840ce not found: ID does not exist" Jul 14 18:18:30.220824 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.220792 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:30.225438 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:30.225417 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-glzrl"] Jul 14 18:18:31.296736 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.296690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:31.392433 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.392400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:31.798263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.798230 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k"] Jul 14 18:18:31.798585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.798571 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed974897-0ac7-4891-be61-010048135a3b" containerName="limitador" Jul 14 18:18:31.798637 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.798586 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed974897-0ac7-4891-be61-010048135a3b" containerName="limitador" Jul 14 18:18:31.798724 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.798712 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed974897-0ac7-4891-be61-010048135a3b" containerName="limitador" Jul 14 18:18:31.802255 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.802236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:31.805099 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.805077 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 18:18:31.805099 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.805094 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:18:31.805258 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.805079 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 18:18:31.805616 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.805597 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ckf98\"" Jul 14 18:18:31.811604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.811581 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k"] Jul 14 18:18:31.829849 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.829788 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:31.909701 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.909667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-data\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:31.909876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.909748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6xn\" (UniqueName: \"kubernetes.io/projected/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-kube-api-access-jn6xn\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:31.930977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:31.930946 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed974897-0ac7-4891-be61-010048135a3b" path="/var/lib/kubelet/pods/ed974897-0ac7-4891-be61-010048135a3b/volumes" Jul 14 18:18:32.010966 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.010931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-data\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:32.011108 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.010997 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6xn\" (UniqueName: \"kubernetes.io/projected/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-kube-api-access-jn6xn\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:32.011285 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.011265 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-data\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:32.019687 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.019663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6xn\" (UniqueName: \"kubernetes.io/projected/ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c-kube-api-access-jn6xn\") pod \"postgres-69fb4594cf-v8l6k\" (UID: \"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:32.113135 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.113056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:32.245497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:32.245470 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k"] Jul 14 18:18:32.248040 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:18:32.248007 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3bc6b5_5d29_4924_80f8_c0b7a5844d6c.slice/crio-08d1f7eae74782e20b034bc936dc203bcb64f1835f0e5e5decd9c2b5e2f8ba49 WatchSource:0}: Error finding container 08d1f7eae74782e20b034bc936dc203bcb64f1835f0e5e5decd9c2b5e2f8ba49: Status 404 returned error can't find the container with id 08d1f7eae74782e20b034bc936dc203bcb64f1835f0e5e5decd9c2b5e2f8ba49 Jul 14 18:18:33.219938 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:33.219906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" event={"ID":"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c","Type":"ContainerStarted","Data":"08d1f7eae74782e20b034bc936dc203bcb64f1835f0e5e5decd9c2b5e2f8ba49"} Jul 14 18:18:37.788308 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:37.788285 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 18:18:38.249568 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:38.249529 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" event={"ID":"ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c","Type":"ContainerStarted","Data":"c6f535834fb7e1d5b60e8d8608ff870f7c6abde2096e5db85a648098114df133"} Jul 14 18:18:38.249755 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:38.249629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:38.269150 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:38.269095 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" podStartSLOduration=1.732687119 podStartE2EDuration="7.269079365s" totalImagesPullingTime="5.536392246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:38.268212967 +0000 UTC m=+483.654111419" watchObservedRunningTime="2026-07-14 18:18:38.269079365 +0000 UTC m=+483.654977818" Jul 14 18:18:44.286209 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:44.286178 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-v8l6k" Jul 14 18:18:44.636395 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:44.636313 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:45.129406 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:45.129372 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:51.873720 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.873686 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:18:51.895180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.895148 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:18:51.895337 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.895248 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:51.897439 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.897414 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 18:18:51.897561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.897440 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 18:18:51.906051 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:51.897972 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xmlkp\"" Jul 14 18:18:52.000861 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.000794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.001054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.000875 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67cw\" (UniqueName: \"kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.034672 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.034627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:52.041556 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.041528 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:18:52.044447 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.044428 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.059138 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.059107 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:18:52.102227 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.102189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.102402 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.102242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67cw\" (UniqueName: \"kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.104622 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.104596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.112675 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.112649 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67cw\" (UniqueName: \"kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw\") pod \"maas-controller-9cbf4c7d4-q9k6k\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.135577 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.135493 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:52.202755 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.202715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdx6g\" (UniqueName: \"kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.202940 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.202769 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.213118 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.213092 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:52.233409 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.233381 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:52.304298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.304267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdx6g\" (UniqueName: \"kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.304441 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.304310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.306718 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.306692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.316741 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.316714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdx6g\" (UniqueName: \"kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g\") pod \"maas-controller-7dcd7b898f-s8m6x\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.348216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.348171 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:18:52.350293 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:18:52.350263 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da380bb_7274_4ac6_ade2_960bde75d7ea.slice/crio-92d73a040e2176455cabbb4c8121fd089b0be7a284ae45571d3a460911bff869 WatchSource:0}: Error finding container 92d73a040e2176455cabbb4c8121fd089b0be7a284ae45571d3a460911bff869: Status 404 returned error can't find the container with id 92d73a040e2176455cabbb4c8121fd089b0be7a284ae45571d3a460911bff869 Jul 14 18:18:52.354400 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.354295 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:52.491405 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:52.491371 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:18:52.494492 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:18:52.494460 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ed7fb2_79c8_48eb_9a9d_1994b30ae9dd.slice/crio-03861e09bf5c041def6b06166937cd101a43bfad4f0963397f434543b5ae56a1 WatchSource:0}: Error finding container 03861e09bf5c041def6b06166937cd101a43bfad4f0963397f434543b5ae56a1: Status 404 returned error can't find the container with id 03861e09bf5c041def6b06166937cd101a43bfad4f0963397f434543b5ae56a1 Jul 14 18:18:53.327984 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:53.327945 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" event={"ID":"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd","Type":"ContainerStarted","Data":"03861e09bf5c041def6b06166937cd101a43bfad4f0963397f434543b5ae56a1"} Jul 14 18:18:53.329609 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:53.329542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" event={"ID":"1da380bb-7274-4ac6-ade2-960bde75d7ea","Type":"ContainerStarted","Data":"92d73a040e2176455cabbb4c8121fd089b0be7a284ae45571d3a460911bff869"} Jul 14 18:18:54.914778 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:54.914740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:55.013874 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.013835 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:18:55.341223 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.341127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" event={"ID":"1da380bb-7274-4ac6-ade2-960bde75d7ea","Type":"ContainerStarted","Data":"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606"} Jul 14 18:18:55.341402 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.341283 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:18:55.342541 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.342515 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" event={"ID":"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd","Type":"ContainerStarted","Data":"d32e3a98eaebc4da1bf1898898de4920e74495a65363a7cc7eef3a9353720d8a"} Jul 14 18:18:55.342659 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.342630 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:18:55.392844 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.392778 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" podStartSLOduration=2.049450592 podStartE2EDuration="4.392761321s" totalImagesPullingTime="2.343310729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:55.391299506 +0000 UTC m=+500.777197959" watchObservedRunningTime="2026-07-14 18:18:55.392761321 +0000 UTC m=+500.778659774" Jul 14 18:18:55.429285 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:18:55.429221 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" podStartSLOduration=1.220091896 podStartE2EDuration="3.429201727s" totalImagesPullingTime="2.209109831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:18:55.428534172 +0000 UTC m=+500.814432650" watchObservedRunningTime="2026-07-14 18:18:55.429201727 +0000 UTC m=+500.815100181" Jul 14 18:19:06.349359 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.349323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:19:06.349765 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.349403 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:19:06.482943 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.482902 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:19:06.483203 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.483146 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" podUID="1da380bb-7274-4ac6-ade2-960bde75d7ea" containerName="manager" containerID="cri-o://7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606" gracePeriod=30 Jul 14 18:19:06.725847 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.725823 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:19:06.833894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.833859 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w67cw\" (UniqueName: \"kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw\") pod \"1da380bb-7274-4ac6-ade2-960bde75d7ea\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " Jul 14 18:19:06.834069 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.833944 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert\") pod \"1da380bb-7274-4ac6-ade2-960bde75d7ea\" (UID: \"1da380bb-7274-4ac6-ade2-960bde75d7ea\") " Jul 14 18:19:06.836113 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.836076 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1da380bb-7274-4ac6-ade2-960bde75d7ea" (UID: "1da380bb-7274-4ac6-ade2-960bde75d7ea"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:19:06.836234 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.836141 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw" (OuterVolumeSpecName: "kube-api-access-w67cw") pod "1da380bb-7274-4ac6-ade2-960bde75d7ea" (UID: "1da380bb-7274-4ac6-ade2-960bde75d7ea"). InnerVolumeSpecName "kube-api-access-w67cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:19:06.935284 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.935188 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1da380bb-7274-4ac6-ade2-960bde75d7ea-webhook-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:19:06.935284 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:06.935224 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67cw\" (UniqueName: \"kubernetes.io/projected/1da380bb-7274-4ac6-ade2-960bde75d7ea-kube-api-access-w67cw\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:19:07.405636 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.405600 2567 generic.go:356] "Generic (PLEG): container finished" podID="1da380bb-7274-4ac6-ade2-960bde75d7ea" containerID="7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606" exitCode=0 Jul 14 18:19:07.406060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.405641 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" Jul 14 18:19:07.406060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.405688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" event={"ID":"1da380bb-7274-4ac6-ade2-960bde75d7ea","Type":"ContainerDied","Data":"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606"} Jul 14 18:19:07.406060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.405738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cbf4c7d4-q9k6k" event={"ID":"1da380bb-7274-4ac6-ade2-960bde75d7ea","Type":"ContainerDied","Data":"92d73a040e2176455cabbb4c8121fd089b0be7a284ae45571d3a460911bff869"} Jul 14 18:19:07.406060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.405761 2567 scope.go:117] "RemoveContainer" containerID="7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606" Jul 14 18:19:07.417249 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.416951 2567 scope.go:117] "RemoveContainer" containerID="7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606" Jul 14 18:19:07.417677 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:19:07.417655 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606\": container with ID starting with 7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606 not found: ID does not exist" containerID="7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606" Jul 14 18:19:07.417746 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.417685 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606"} err="failed to get container status \"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606\": rpc error: code = NotFound desc = could not find container \"7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606\": container with ID starting with 7bc6596b5e12024dddb16f8a128bd4aa6223fdc068b501735c93aa56413ab606 not found: ID does not exist" Jul 14 18:19:07.432357 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.432323 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:19:07.443441 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.443417 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cbf4c7d4-q9k6k"] Jul 14 18:19:07.930911 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:07.930878 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da380bb-7274-4ac6-ade2-960bde75d7ea" path="/var/lib/kubelet/pods/1da380bb-7274-4ac6-ade2-960bde75d7ea/volumes" Jul 14 18:19:27.117382 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:27.117298 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:27.216845 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:27.216797 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:27.814922 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:27.814887 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:28.217430 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.217347 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g"] Jul 14 18:19:28.217788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.217721 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da380bb-7274-4ac6-ade2-960bde75d7ea" containerName="manager" Jul 14 18:19:28.217788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.217732 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da380bb-7274-4ac6-ade2-960bde75d7ea" containerName="manager" Jul 14 18:19:28.217788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.217784 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da380bb-7274-4ac6-ade2-960bde75d7ea" containerName="manager" Jul 14 18:19:28.219949 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.219932 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.222515 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.222496 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 14 18:19:28.222587 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.222496 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lmq6m\"" Jul 14 18:19:28.238318 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.238287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g"] Jul 14 18:19:28.255497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.255470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:28.334277 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.334246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvls9\" (UniqueName: \"kubernetes.io/projected/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-kube-api-access-nvls9\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.334452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.334315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-maas-api-tls\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.334452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.334337 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-openshift-service-ca\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.435311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.435276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-maas-api-tls\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.435311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.435319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-openshift-service-ca\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.435517 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.435368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvls9\" (UniqueName: \"kubernetes.io/projected/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-kube-api-access-nvls9\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.436019 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.435995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-openshift-service-ca\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.437709 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.437683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-maas-api-tls\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.445748 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.445724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvls9\" (UniqueName: \"kubernetes.io/projected/0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0-kube-api-access-nvls9\") pod \"maas-api-54b45ff5dc-gmz2g\" (UID: \"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0\") " pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.529666 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.529636 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:28.674338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.674307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g"] Jul 14 18:19:28.677460 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:19:28.677429 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e34e5cf_0cce_45a2_9969_1ba3ad56f1c0.slice/crio-667089c2c41560362c57a3e723ffe48bd214d9cbf422abaec167cf739ec443b0 WatchSource:0}: Error finding container 667089c2c41560362c57a3e723ffe48bd214d9cbf422abaec167cf739ec443b0: Status 404 returned error can't find the container with id 667089c2c41560362c57a3e723ffe48bd214d9cbf422abaec167cf739ec443b0 Jul 14 18:19:28.782609 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.782506 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:28.887772 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.887728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:28.942154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.942114 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s"] Jul 14 18:19:28.945630 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.945608 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:28.947929 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.947903 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 18:19:28.961959 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:28.961935 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s"] Jul 14 18:19:29.041888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.041776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89q2s\" (UniqueName: \"kubernetes.io/projected/79ccee40-6189-4489-8087-a3dc68495c21-kube-api-access-89q2s\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.041888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.041850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ccee40-6189-4489-8087-a3dc68495c21-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.080498 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.080464 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-gfmmb"] Jul 14 18:19:29.083189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.083172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.085339 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.085321 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sh7mq\"" Jul 14 18:19:29.094432 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.094404 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-gfmmb"] Jul 14 18:19:29.142444 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.142407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89q2s\" (UniqueName: \"kubernetes.io/projected/79ccee40-6189-4489-8087-a3dc68495c21-kube-api-access-89q2s\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.142620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.142494 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ccee40-6189-4489-8087-a3dc68495c21-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.143162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.143143 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79ccee40-6189-4489-8087-a3dc68495c21-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.152170 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.152147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89q2s\" (UniqueName: \"kubernetes.io/projected/79ccee40-6189-4489-8087-a3dc68495c21-kube-api-access-89q2s\") pod \"payload-pre-processing-74584bb9f8-4dk9s\" (UID: \"79ccee40-6189-4489-8087-a3dc68495c21\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.243309 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.243267 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b98be0-12b3-4486-a0ce-ca7627424ecb-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.243749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.243400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4pdr\" (UniqueName: \"kubernetes.io/projected/29b98be0-12b3-4486-a0ce-ca7627424ecb-kube-api-access-l4pdr\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.256180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.256151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:29.344862 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.344796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b98be0-12b3-4486-a0ce-ca7627424ecb-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.345030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.344927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4pdr\" (UniqueName: \"kubernetes.io/projected/29b98be0-12b3-4486-a0ce-ca7627424ecb-kube-api-access-l4pdr\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.345682 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.345653 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b98be0-12b3-4486-a0ce-ca7627424ecb-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.355196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.355161 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4pdr\" (UniqueName: \"kubernetes.io/projected/29b98be0-12b3-4486-a0ce-ca7627424ecb-kube-api-access-l4pdr\") pod \"payload-processing-f9b8d9b9b-gfmmb\" (UID: \"29b98be0-12b3-4486-a0ce-ca7627424ecb\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.393425 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.393388 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:29.430536 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.430497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s"] Jul 14 18:19:29.434488 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:19:29.434454 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ccee40_6189_4489_8087_a3dc68495c21.slice/crio-bab89376230e3481d9389b3465b2c59c47f1505e59e042382dda8e7193d11c91 WatchSource:0}: Error finding container bab89376230e3481d9389b3465b2c59c47f1505e59e042382dda8e7193d11c91: Status 404 returned error can't find the container with id bab89376230e3481d9389b3465b2c59c47f1505e59e042382dda8e7193d11c91 Jul 14 18:19:29.512145 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.512112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" event={"ID":"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0","Type":"ContainerStarted","Data":"667089c2c41560362c57a3e723ffe48bd214d9cbf422abaec167cf739ec443b0"} Jul 14 18:19:29.513803 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.513770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" event={"ID":"79ccee40-6189-4489-8087-a3dc68495c21","Type":"ContainerStarted","Data":"bab89376230e3481d9389b3465b2c59c47f1505e59e042382dda8e7193d11c91"} Jul 14 18:19:29.541330 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:29.541297 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-gfmmb"] Jul 14 18:19:29.544328 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:19:29.544296 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b98be0_12b3_4486_a0ce_ca7627424ecb.slice/crio-94a0b8284dde97bcf675da3a6a6b4389830d36df94e80b2a194c70c479cb919c WatchSource:0}: Error finding container 94a0b8284dde97bcf675da3a6a6b4389830d36df94e80b2a194c70c479cb919c: Status 404 returned error can't find the container with id 94a0b8284dde97bcf675da3a6a6b4389830d36df94e80b2a194c70c479cb919c Jul 14 18:19:30.523215 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:30.523176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" event={"ID":"0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0","Type":"ContainerStarted","Data":"030136754d26bc46581313a6ecca42185dc694dd87723773e26d5fabc7a549ac"} Jul 14 18:19:30.523650 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:30.523259 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:30.524979 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:30.524951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" event={"ID":"29b98be0-12b3-4486-a0ce-ca7627424ecb","Type":"ContainerStarted","Data":"94a0b8284dde97bcf675da3a6a6b4389830d36df94e80b2a194c70c479cb919c"} Jul 14 18:19:30.549408 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:30.549302 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" podStartSLOduration=1.938453284 podStartE2EDuration="3.549281316s" totalImagesPullingTime="1.610828032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:30.547596671 +0000 UTC m=+535.933495124" watchObservedRunningTime="2026-07-14 18:19:30.549281316 +0000 UTC m=+535.935179768" Jul 14 18:19:31.531784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:31.531746 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" event={"ID":"79ccee40-6189-4489-8087-a3dc68495c21","Type":"ContainerStarted","Data":"d60c63a071fdcaaf6589571c8c7c82ec51fce4ff6c35d09a569907683c50492d"} Jul 14 18:19:31.532235 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:31.531988 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:31.556918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:31.556873 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" podStartSLOduration=1.7740566389999999 podStartE2EDuration="3.556851941s" totalImagesPullingTime="1.782795302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:31.555718771 +0000 UTC m=+536.941617224" watchObservedRunningTime="2026-07-14 18:19:31.556851941 +0000 UTC m=+536.942750393" Jul 14 18:19:32.538216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:32.538165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" event={"ID":"29b98be0-12b3-4486-a0ce-ca7627424ecb","Type":"ContainerStarted","Data":"1e01de75172316bb52a3b0d0451a3772972ae8abda0859799ffd12b0b0f0a6a2"} Jul 14 18:19:32.538666 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:32.538525 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:32.570696 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:32.570631 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" podStartSLOduration=1.018206309 podStartE2EDuration="3.570613033s" totalImagesPullingTime="2.552406724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:32.569057203 +0000 UTC m=+537.954955676" watchObservedRunningTime="2026-07-14 18:19:32.570613033 +0000 UTC m=+537.956511543" Jul 14 18:19:36.536764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:36.536736 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b45ff5dc-gmz2g" Jul 14 18:19:38.475915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:38.475869 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:38.574298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:38.574265 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:42.539800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:42.539770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-4dk9s" Jul 14 18:19:43.545622 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:43.545589 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-gfmmb" Jul 14 18:19:47.079504 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.079473 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k"] Jul 14 18:19:47.084794 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.084772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.088492 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.088469 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6s2bs\"" Jul 14 18:19:47.088914 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.088894 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 14 18:19:47.089301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.089286 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:19:47.092601 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.092585 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:19:47.111585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.111559 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k"] Jul 14 18:19:47.113545 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.113742 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.113888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.113888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.113888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113839 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrswr\" (UniqueName: \"kubernetes.io/projected/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-kube-api-access-hrswr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.114062 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.113927 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.114326 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.114294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:47.214457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.214653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.214653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.214653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.214653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.214653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrswr\" (UniqueName: \"kubernetes.io/projected/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-kube-api-access-hrswr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.215048 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.214990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.215048 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.215008 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.215160 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.215091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.216803 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.216777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.217055 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.217039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.225613 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.225586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrswr\" (UniqueName: \"kubernetes.io/projected/a58e0ed6-b9e1-41a0-9680-2f5a39b58c57-kube-api-access-hrswr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k\" (UID: \"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.395670 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.395585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:47.536990 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.536795 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k"] Jul 14 18:19:47.539880 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:19:47.539845 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e0ed6_b9e1_41a0_9680_2f5a39b58c57.slice/crio-e2e5119146aa42235bbfa33f11e9cd5473f0e6d86be728a69485a799f1d958f5 WatchSource:0}: Error finding container e2e5119146aa42235bbfa33f11e9cd5473f0e6d86be728a69485a799f1d958f5: Status 404 returned error can't find the container with id e2e5119146aa42235bbfa33f11e9cd5473f0e6d86be728a69485a799f1d958f5 Jul 14 18:19:47.615884 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:47.615841 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" event={"ID":"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57","Type":"ContainerStarted","Data":"e2e5119146aa42235bbfa33f11e9cd5473f0e6d86be728a69485a799f1d958f5"} Jul 14 18:19:51.049068 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:51.049028 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:56.666888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:56.666852 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" event={"ID":"a58e0ed6-b9e1-41a0-9680-2f5a39b58c57","Type":"ContainerStarted","Data":"3fcccb70d7100f7824f9d8f4cc9bc3eca2f47ae337386c21b3214b01ee9d52a8"} Jul 14 18:19:56.691559 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:56.691503 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" podStartSLOduration=1.007839133 podStartE2EDuration="9.691482593s" totalImagesPullingTime="8.68364346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:19:56.690938632 +0000 UTC m=+562.076837085" watchObservedRunningTime="2026-07-14 18:19:56.691482593 +0000 UTC m=+562.077381047" Jul 14 18:19:57.107768 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:57.107716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:19:57.396686 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:57.396600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:57.396686 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:57.396680 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:57.409099 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:57.409076 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:19:57.682910 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:19:57.682823 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k" Jul 14 18:20:04.998266 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:04.998223 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:10.947867 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:10.947830 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9"] Jul 14 18:20:10.952152 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:10.952122 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:10.954529 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:10.954506 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 18:20:10.961485 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:10.961460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9"] Jul 14 18:20:10.974426 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:10.974393 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:11.039838 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.039782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.039838 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.039842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.040101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.039948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.040101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.040016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp6b\" (UniqueName: \"kubernetes.io/projected/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-kube-api-access-ltp6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.040101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.040065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.040250 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.040158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141170 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141131 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141224 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141279 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp6b\" (UniqueName: \"kubernetes.io/projected/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-kube-api-access-ltp6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141736 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141702 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141736 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.141907 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.141762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.143501 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.143472 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.143806 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.143785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.150891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.150866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp6b\" (UniqueName: \"kubernetes.io/projected/f00d4bf2-0c86-4825-bc4a-8d1b00213d38-kube-api-access-ltp6b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9\" (UID: \"f00d4bf2-0c86-4825-bc4a-8d1b00213d38\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.263303 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.263266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:11.420180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.420077 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9"] Jul 14 18:20:11.422644 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:20:11.422606 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00d4bf2_0c86_4825_bc4a_8d1b00213d38.slice/crio-9776167214e6a53c7e7fe8139939392655ddb3fae8a028afc4b26dcadb637614 WatchSource:0}: Error finding container 9776167214e6a53c7e7fe8139939392655ddb3fae8a028afc4b26dcadb637614: Status 404 returned error can't find the container with id 9776167214e6a53c7e7fe8139939392655ddb3fae8a028afc4b26dcadb637614 Jul 14 18:20:11.738280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.738246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" event={"ID":"f00d4bf2-0c86-4825-bc4a-8d1b00213d38","Type":"ContainerStarted","Data":"655d350e358c5a314055e8c654e8f0c8569c3a6a935f14912800d6d5e119ad84"} Jul 14 18:20:11.738456 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.738302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" event={"ID":"f00d4bf2-0c86-4825-bc4a-8d1b00213d38","Type":"ContainerStarted","Data":"9776167214e6a53c7e7fe8139939392655ddb3fae8a028afc4b26dcadb637614"} Jul 14 18:20:11.758361 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:11.758306 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" podStartSLOduration=1.567292455 podStartE2EDuration="1.758290827s" totalImagesPullingTime="190.998372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:20:11.756629052 +0000 UTC m=+577.142527506" watchObservedRunningTime="2026-07-14 18:20:11.758290827 +0000 UTC m=+577.144189280" Jul 14 18:20:16.292935 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:16.292889 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:16.388376 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:16.388341 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:21.263892 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:21.263853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:21.264372 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:21.263935 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:21.276301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:21.276274 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:21.802598 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:21.802568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9" Jul 14 18:20:21.934762 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:21.934710 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:25.814077 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:25.814040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:31.902441 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:31.902376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:20:35.941920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:35.941778 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:20:35.945075 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:35.943202 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:20:52.587068 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:20:52.587013 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:21:20.151482 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:21:20.151450 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:21:37.218925 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:21:37.218889 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:21:37.319915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:21:37.319867 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:21:38.616472 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:21:38.616441 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:21:38.703841 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:21:38.703783 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:03.267135 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:03.267101 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:03.432498 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:03.432466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:47.921578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:47.921542 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:52.343418 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:52.343383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.277929 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.277890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.395006 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.394968 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:22:54.395232 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.395210 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" podUID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" containerName="manager" containerID="cri-o://d32e3a98eaebc4da1bf1898898de4920e74495a65363a7cc7eef3a9353720d8a" gracePeriod=30 Jul 14 18:22:54.533216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.533133 2567 generic.go:356] "Generic (PLEG): container finished" podID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" containerID="d32e3a98eaebc4da1bf1898898de4920e74495a65363a7cc7eef3a9353720d8a" exitCode=0 Jul 14 18:22:54.533216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.533195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" event={"ID":"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd","Type":"ContainerDied","Data":"d32e3a98eaebc4da1bf1898898de4920e74495a65363a7cc7eef3a9353720d8a"} Jul 14 18:22:54.646145 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.646120 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:22:54.781593 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.781535 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert\") pod \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " Jul 14 18:22:54.781778 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.781673 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdx6g\" (UniqueName: \"kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g\") pod \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\" (UID: \"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd\") " Jul 14 18:22:54.783782 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.783716 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g" (OuterVolumeSpecName: "kube-api-access-cdx6g") pod "67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" (UID: "67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd"). InnerVolumeSpecName "kube-api-access-cdx6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:22:54.783782 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.783744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" (UID: "67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:22:54.883243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.883207 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-webhook-cert\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:22:54.883243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:54.883236 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdx6g\" (UniqueName: \"kubernetes.io/projected/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd-kube-api-access-cdx6g\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:22:55.539245 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.539209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" event={"ID":"67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd","Type":"ContainerDied","Data":"03861e09bf5c041def6b06166937cd101a43bfad4f0963397f434543b5ae56a1"} Jul 14 18:22:55.539245 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.539234 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-s8m6x" Jul 14 18:22:55.539742 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.539265 2567 scope.go:117] "RemoveContainer" containerID="d32e3a98eaebc4da1bf1898898de4920e74495a65363a7cc7eef3a9353720d8a" Jul 14 18:22:55.565541 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.565512 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:22:55.570063 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.570033 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-s8m6x"] Jul 14 18:22:55.930676 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:55.930601 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" path="/var/lib/kubelet/pods/67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd/volumes" Jul 14 18:22:56.275932 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.275900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.402186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.402153 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-nqprv"] Jul 14 18:22:56.402571 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.402559 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" containerName="manager" Jul 14 18:22:56.402744 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.402572 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" containerName="manager" Jul 14 18:22:56.402744 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.402627 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ed7fb2-79c8-48eb-9a9d-1994b30ae9dd" containerName="manager" Jul 14 18:22:56.405640 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.405624 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.408000 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.407974 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 18:22:56.408540 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.408522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xmlkp\"" Jul 14 18:22:56.408638 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.408542 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 18:22:56.414533 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.414510 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-nqprv"] Jul 14 18:22:56.500884 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.500843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82ff\" (UniqueName: \"kubernetes.io/projected/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-kube-api-access-k82ff\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.501059 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.500893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-webhook-cert\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.601920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.601825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82ff\" (UniqueName: \"kubernetes.io/projected/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-kube-api-access-k82ff\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.601920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.601873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-webhook-cert\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.604310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.604290 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-webhook-cert\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.613671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.613644 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82ff\" (UniqueName: \"kubernetes.io/projected/63c8ac52-0bb3-43a1-8521-cea6f8d6d661-kube-api-access-k82ff\") pod \"maas-controller-7dcd7b898f-nqprv\" (UID: \"63c8ac52-0bb3-43a1-8521-cea6f8d6d661\") " pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.716876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.716834 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:56.856690 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.856664 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcd7b898f-nqprv"] Jul 14 18:22:56.858944 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:22:56.858916 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c8ac52_0bb3_43a1_8521_cea6f8d6d661.slice/crio-7d557f47324370953b9d9bf1ed63e1e1f4e235703e49065532e0329696ec27d0 WatchSource:0}: Error finding container 7d557f47324370953b9d9bf1ed63e1e1f4e235703e49065532e0329696ec27d0: Status 404 returned error can't find the container with id 7d557f47324370953b9d9bf1ed63e1e1f4e235703e49065532e0329696ec27d0 Jul 14 18:22:56.860430 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:56.860411 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:22:57.551710 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:57.551676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" event={"ID":"63c8ac52-0bb3-43a1-8521-cea6f8d6d661","Type":"ContainerStarted","Data":"ea054586c08b1b8557e8c12bf7fa08ca4e1e04100607750c6a0b667399fd8726"} Jul 14 18:22:57.551930 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:57.551734 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" event={"ID":"63c8ac52-0bb3-43a1-8521-cea6f8d6d661","Type":"ContainerStarted","Data":"7d557f47324370953b9d9bf1ed63e1e1f4e235703e49065532e0329696ec27d0"} Jul 14 18:22:57.551930 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:57.551840 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:22:57.573019 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:22:57.572969 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" podStartSLOduration=1.021069785 podStartE2EDuration="1.572953146s" totalImagesPullingTime="551.883361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:57.570656179 +0000 UTC m=+742.956554623" watchObservedRunningTime="2026-07-14 18:22:57.572953146 +0000 UTC m=+742.958851622" Jul 14 18:23:07.047110 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:07.047070 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:08.559539 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:08.559510 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcd7b898f-nqprv" Jul 14 18:23:35.226532 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:35.226500 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:35.326780 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:35.326750 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:35.470755 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:35.470727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:41.909588 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:41.909550 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:41.971361 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:41.971324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:42.057408 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:42.057361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:42.157467 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:42.157430 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:42.291490 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:42.291457 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:49.166251 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:49.166213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:49.229021 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:23:49.228989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:23.088062 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:24:23.087987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:35.973385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:25:35.973268 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:25:35.977045 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:25:35.975590 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:27:23.503424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:23.503340 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:37.345655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.345617 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd775db8c-7bszs"] Jul 14 18:27:37.348426 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.348406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.352036 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.352012 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 18:27:37.352147 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.352083 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 18:27:37.353634 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.353606 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 18:27:37.353741 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.353694 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 18:27:37.353804 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.353738 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5ljq\"" Jul 14 18:27:37.353934 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.353917 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 18:27:37.354096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.354074 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 18:27:37.354538 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.354361 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 18:27:37.357257 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.357236 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 18:27:37.370783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.370758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd775db8c-7bszs"] Jul 14 18:27:37.406606 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.406576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:37.451431 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451390 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451431 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-service-ca\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451627 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-oauth-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451627 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-oauth-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451627 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-console-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451729 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-trusted-ca-bundle\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.451729 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.451674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctqg\" (UniqueName: \"kubernetes.io/projected/02d7861e-996b-435a-ba4a-7a003620b178-kube-api-access-pctqg\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.552887 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.552854 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctqg\" (UniqueName: \"kubernetes.io/projected/02d7861e-996b-435a-ba4a-7a003620b178-kube-api-access-pctqg\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.552941 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.552961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-service-ca\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.552999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-oauth-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553054 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-oauth-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553254 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-console-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553254 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-trusted-ca-bundle\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553724 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-service-ca\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553838 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-oauth-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.553920 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.553899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-console-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.554533 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.554512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02d7861e-996b-435a-ba4a-7a003620b178-trusted-ca-bundle\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.555480 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.555452 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-oauth-config\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.555588 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.555572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02d7861e-996b-435a-ba4a-7a003620b178-console-serving-cert\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.566574 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.566553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctqg\" (UniqueName: \"kubernetes.io/projected/02d7861e-996b-435a-ba4a-7a003620b178-kube-api-access-pctqg\") pod \"console-6fd775db8c-7bszs\" (UID: \"02d7861e-996b-435a-ba4a-7a003620b178\") " pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.567620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.567595 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:37.658671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.658577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:37.840505 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.840474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd775db8c-7bszs"] Jul 14 18:27:37.843648 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:27:37.843621 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d7861e_996b_435a_ba4a_7a003620b178.slice/crio-1707ce59e1048c30b5c2872553ef063fdbacc25a9baa06e54633a685f39351ff WatchSource:0}: Error finding container 1707ce59e1048c30b5c2872553ef063fdbacc25a9baa06e54633a685f39351ff: Status 404 returned error can't find the container with id 1707ce59e1048c30b5c2872553ef063fdbacc25a9baa06e54633a685f39351ff Jul 14 18:27:37.931753 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.931663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd775db8c-7bszs" event={"ID":"02d7861e-996b-435a-ba4a-7a003620b178","Type":"ContainerStarted","Data":"31cb3a0417162c92ec035f006ef1fd7447e82033a704b191643a1cb3b62213f4"} Jul 14 18:27:37.931753 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:37.931716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd775db8c-7bszs" event={"ID":"02d7861e-996b-435a-ba4a-7a003620b178","Type":"ContainerStarted","Data":"1707ce59e1048c30b5c2872553ef063fdbacc25a9baa06e54633a685f39351ff"} Jul 14 18:27:38.000730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:38.000678 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd775db8c-7bszs" podStartSLOduration=1.000661635 podStartE2EDuration="1.000661635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:38.000301 +0000 UTC m=+1023.386199474" watchObservedRunningTime="2026-07-14 18:27:38.000661635 +0000 UTC m=+1023.386560090" Jul 14 18:27:47.659567 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:47.659463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:47.659567 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:47.659531 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:47.664505 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:47.664478 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:47.984149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:47.984115 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd775db8c-7bszs" Jul 14 18:27:48.188462 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:48.188428 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:48.287298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:48.287199 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:48.356371 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:27:48.356342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:17.814623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:29:17.814590 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:19.421243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:29:19.421201 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:00.152919 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.152835 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb"] Jul 14 18:30:00.156310 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.156291 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" Jul 14 18:30:00.158404 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.158382 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9jplk\"" Jul 14 18:30:00.165182 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.165156 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb"] Jul 14 18:30:00.167727 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.167705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" Jul 14 18:30:00.311301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.311264 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb"] Jul 14 18:30:00.314782 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:30:00.314759 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c11edfe_f183_491a_9451_40418e15993b.slice/crio-3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf WatchSource:0}: Error finding container 3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf: Status 404 returned error can't find the container with id 3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf Jul 14 18:30:00.316706 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.316691 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:30:00.621835 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:00.621774 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" event={"ID":"0c11edfe-f183-491a-9451-40418e15993b","Type":"ContainerStarted","Data":"3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf"} Jul 14 18:30:09.670668 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:09.670573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" event={"ID":"0c11edfe-f183-491a-9451-40418e15993b","Type":"ContainerStarted","Data":"991d8eae0720d9d90fdd6aa9fe264a6eade2bbfebc2330c46ffa0bfcb3890e7e"} Jul 14 18:30:09.688447 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:09.688393 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" podStartSLOduration=1.031741568 podStartE2EDuration="9.688377931s" totalImagesPullingTime="8.656636363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:09.686434079 +0000 UTC m=+1175.072332545" watchObservedRunningTime="2026-07-14 18:30:09.688377931 +0000 UTC m=+1175.074276384" Jul 14 18:30:10.677070 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:10.677037 2567 generic.go:356] "Generic (PLEG): container finished" podID="0c11edfe-f183-491a-9451-40418e15993b" containerID="991d8eae0720d9d90fdd6aa9fe264a6eade2bbfebc2330c46ffa0bfcb3890e7e" exitCode=0 Jul 14 18:30:10.677457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:10.677124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" event={"ID":"0c11edfe-f183-491a-9451-40418e15993b","Type":"ContainerDied","Data":"991d8eae0720d9d90fdd6aa9fe264a6eade2bbfebc2330c46ffa0bfcb3890e7e"} Jul 14 18:30:11.814603 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:11.814575 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" Jul 14 18:30:12.689853 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:12.689806 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" event={"ID":"0c11edfe-f183-491a-9451-40418e15993b","Type":"ContainerDied","Data":"3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf"} Jul 14 18:30:12.690057 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:12.689874 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd6289d92ca7f387e8d91efe6c3ad1a2e4fca347acb17557f24b270b5bbccdf" Jul 14 18:30:12.690057 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:12.689824 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734230-97ghb" Jul 14 18:30:26.345502 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:26.345460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:36.007861 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:36.007830 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:30:36.009615 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:36.009575 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:30:53.892774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:30:53.892729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:33:06.557052 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:33:06.556969 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:33.626086 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:33.625982 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:43.066880 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:43.066837 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:43.169389 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:43.169323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:43.457570 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:43.457469 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:51.728792 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:51.728744 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:56.979865 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:34:56.979795 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:00.219953 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:00.219914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:00.320045 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:00.320009 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:00.721681 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:00.721621 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:07.705431 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:07.705394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:07.806733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:07.806701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:09.141477 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.141433 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:09.142135 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.142116 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c11edfe-f183-491a-9451-40418e15993b" containerName="cleanup" Jul 14 18:35:09.142202 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.142138 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c11edfe-f183-491a-9451-40418e15993b" containerName="cleanup" Jul 14 18:35:09.142243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.142216 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c11edfe-f183-491a-9451-40418e15993b" containerName="cleanup" Jul 14 18:35:09.144452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.144437 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.146619 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.146599 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f37aedf1-serving-cert\"" Jul 14 18:35:09.155917 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.155888 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:09.333634 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.333594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.333634 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.333635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxdv\" (UniqueName: \"kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.333948 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.333746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.435471 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.435380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.435471 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.435420 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxdv\" (UniqueName: \"kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.435710 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.435487 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.435710 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:35:09.435536 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-serving-cert: secret "maas-api-e2e-ait-f37aedf1-serving-cert" not found Jul 14 18:35:09.435710 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:35:09.435615 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls podName:c18db037-0133-4a02-b921-4f71a656dda5 nodeName:}" failed. No retries permitted until 2026-07-14 18:35:09.935598198 +0000 UTC m=+1475.321496644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls") pod "maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" (UID: "c18db037-0133-4a02-b921-4f71a656dda5") : secret "maas-api-e2e-ait-f37aedf1-serving-cert" not found Jul 14 18:35:09.436259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.436236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.450165 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.450134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxdv\" (UniqueName: \"kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.694003 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.693909 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:09.796706 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.796649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:09.893646 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.893602 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:09.941333 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.941299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.943671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.943653 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") pod \"maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:09.994617 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:09.994578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:10.057122 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:10.057085 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:10.203222 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:10.203185 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:10.209407 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:35:10.209377 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18db037_0133_4a02_b921_4f71a656dda5.slice/crio-2c93d70f32ff30eadb0b098d13bcc867a253f825961b00e67ab0188953958fd7 WatchSource:0}: Error finding container 2c93d70f32ff30eadb0b098d13bcc867a253f825961b00e67ab0188953958fd7: Status 404 returned error can't find the container with id 2c93d70f32ff30eadb0b098d13bcc867a253f825961b00e67ab0188953958fd7 Jul 14 18:35:10.210632 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:10.210609 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:35:11.153493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:11.153396 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" event={"ID":"c18db037-0133-4a02-b921-4f71a656dda5","Type":"ContainerStarted","Data":"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27"} Jul 14 18:35:11.153493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:11.153465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" event={"ID":"c18db037-0133-4a02-b921-4f71a656dda5","Type":"ContainerStarted","Data":"2c93d70f32ff30eadb0b098d13bcc867a253f825961b00e67ab0188953958fd7"} Jul 14 18:35:11.153684 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:11.153607 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:11.175863 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:11.175787 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" podStartSLOduration=1.5811451600000002 podStartE2EDuration="2.175770319s" totalImagesPullingTime="594.625159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:35:11.174084996 +0000 UTC m=+1476.559983450" watchObservedRunningTime="2026-07-14 18:35:11.175770319 +0000 UTC m=+1476.561668831" Jul 14 18:35:11.843283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:11.843249 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:17.163752 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:17.163723 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:24.911478 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:24.911447 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:24.911973 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:24.911631 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" podUID="c18db037-0133-4a02-b921-4f71a656dda5" containerName="maas-api" containerID="cri-o://48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27" gracePeriod=30 Jul 14 18:35:24.933057 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:24.933028 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:25.161279 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.161257 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:25.224784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.224689 2567 generic.go:356] "Generic (PLEG): container finished" podID="c18db037-0133-4a02-b921-4f71a656dda5" containerID="48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27" exitCode=0 Jul 14 18:35:25.224784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.224737 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" Jul 14 18:35:25.224784 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.224756 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" event={"ID":"c18db037-0133-4a02-b921-4f71a656dda5","Type":"ContainerDied","Data":"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27"} Jul 14 18:35:25.225050 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.224799 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4" event={"ID":"c18db037-0133-4a02-b921-4f71a656dda5","Type":"ContainerDied","Data":"2c93d70f32ff30eadb0b098d13bcc867a253f825961b00e67ab0188953958fd7"} Jul 14 18:35:25.225050 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.224837 2567 scope.go:117] "RemoveContainer" containerID="48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27" Jul 14 18:35:25.233329 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.233312 2567 scope.go:117] "RemoveContainer" containerID="48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27" Jul 14 18:35:25.233586 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:35:25.233563 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27\": container with ID starting with 48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27 not found: ID does not exist" containerID="48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27" Jul 14 18:35:25.233653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.233599 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27"} err="failed to get container status \"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27\": rpc error: code = NotFound desc = could not find container \"48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27\": container with ID starting with 48a2af44a850d80b7c40df4d219a78b09d8d4b59cb73853a08a8ab7715551e27 not found: ID does not exist" Jul 14 18:35:25.285383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.285350 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") pod \"c18db037-0133-4a02-b921-4f71a656dda5\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " Jul 14 18:35:25.285578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.285421 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca\") pod \"c18db037-0133-4a02-b921-4f71a656dda5\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " Jul 14 18:35:25.285578 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.285468 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnxdv\" (UniqueName: \"kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv\") pod \"c18db037-0133-4a02-b921-4f71a656dda5\" (UID: \"c18db037-0133-4a02-b921-4f71a656dda5\") " Jul 14 18:35:25.285787 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.285763 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c18db037-0133-4a02-b921-4f71a656dda5" (UID: "c18db037-0133-4a02-b921-4f71a656dda5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:25.287634 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.287611 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c18db037-0133-4a02-b921-4f71a656dda5" (UID: "c18db037-0133-4a02-b921-4f71a656dda5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:35:25.287704 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.287622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv" (OuterVolumeSpecName: "kube-api-access-hnxdv") pod "c18db037-0133-4a02-b921-4f71a656dda5" (UID: "c18db037-0133-4a02-b921-4f71a656dda5"). InnerVolumeSpecName "kube-api-access-hnxdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:35:25.386689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.386657 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c18db037-0133-4a02-b921-4f71a656dda5-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:35:25.386689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.386686 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxdv\" (UniqueName: \"kubernetes.io/projected/c18db037-0133-4a02-b921-4f71a656dda5-kube-api-access-hnxdv\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:35:25.386689 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.386696 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c18db037-0133-4a02-b921-4f71a656dda5-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:35:25.548060 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.548018 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:25.553977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.553947 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f37aedf1-7fb89b7c59-dspq4"] Jul 14 18:35:25.931612 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:25.931532 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c18db037-0133-4a02-b921-4f71a656dda5" path="/var/lib/kubelet/pods/c18db037-0133-4a02-b921-4f71a656dda5/volumes" Jul 14 18:35:29.323457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:29.323424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:36.031699 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:36.031541 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:36.046492 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:36.046472 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:35:36.046692 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:36.046676 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:35:36.091962 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:36.091929 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:39.558783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:39.558755 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:39.959181 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:39.959098 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:40.057269 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:40.057239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:40.184891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:40.184857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:41.428783 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.428735 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:41.539977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.539946 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:35:41.540474 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.540456 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18db037-0133-4a02-b921-4f71a656dda5" containerName="maas-api" Jul 14 18:35:41.540580 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.540478 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18db037-0133-4a02-b921-4f71a656dda5" containerName="maas-api" Jul 14 18:35:41.540642 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.540590 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18db037-0133-4a02-b921-4f71a656dda5" containerName="maas-api" Jul 14 18:35:41.545208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.545188 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.548024 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.547994 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-499cb51a-serving-cert\"" Jul 14 18:35:41.560337 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.560310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:35:41.590244 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.590212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:41.647015 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.646981 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.647196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.647079 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.647196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.647136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkk5\" (UniqueName: \"kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.688320 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.688238 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:41.748179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.748144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.748371 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.748196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkk5\" (UniqueName: \"kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.748371 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.748258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.748914 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.748895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.750519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.750500 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.758014 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.757989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkk5\" (UniqueName: \"kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5\") pod \"maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:41.856569 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:41.856528 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:42.002289 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:42.002262 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:35:42.005425 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:35:42.005397 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef93f3b_6b7d_4682_895e_bb18c7bd33f7.slice/crio-480cdf9c8401047369a9c61180b1dbf0b4ec6e842b82f84015628cc57d6889e8 WatchSource:0}: Error finding container 480cdf9c8401047369a9c61180b1dbf0b4ec6e842b82f84015628cc57d6889e8: Status 404 returned error can't find the container with id 480cdf9c8401047369a9c61180b1dbf0b4ec6e842b82f84015628cc57d6889e8 Jul 14 18:35:42.139123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:42.139085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:42.240390 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:42.240360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:42.312799 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:42.312764 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" event={"ID":"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7","Type":"ContainerStarted","Data":"480cdf9c8401047369a9c61180b1dbf0b4ec6e842b82f84015628cc57d6889e8"} Jul 14 18:35:43.319713 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:43.319675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" event={"ID":"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7","Type":"ContainerStarted","Data":"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b"} Jul 14 18:35:43.320234 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:43.319788 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:43.338977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:43.338924 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" podStartSLOduration=1.98205755 podStartE2EDuration="2.338905173s" totalImagesPullingTime="356.847623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:35:43.338039131 +0000 UTC m=+1508.723937589" watchObservedRunningTime="2026-07-14 18:35:43.338905173 +0000 UTC m=+1508.724803643" Jul 14 18:35:49.330314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:49.330283 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:35:50.876272 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:50.876213 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:50.975237 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:50.975203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:54.208723 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:35:54.208692 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:46.145045 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.145006 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:46.257540 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.257507 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:46.656377 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.656335 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:36:46.656626 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.656589 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" podUID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" containerName="maas-api" containerID="cri-o://199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b" gracePeriod=30 Jul 14 18:36:46.689058 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.688998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:46.904967 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.904946 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:36:46.978981 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.978954 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca\") pod \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " Jul 14 18:36:46.979135 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.979027 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggkk5\" (UniqueName: \"kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5\") pod \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " Jul 14 18:36:46.979135 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.979057 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls\") pod \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\" (UID: \"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7\") " Jul 14 18:36:46.979374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.979346 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" (UID: "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:36:46.981161 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.981138 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5" (OuterVolumeSpecName: "kube-api-access-ggkk5") pod "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" (UID: "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7"). InnerVolumeSpecName "kube-api-access-ggkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:36:46.981271 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:46.981172 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" (UID: "5ef93f3b-6b7d-4682-895e-bb18c7bd33f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:36:47.080604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.080572 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkk5\" (UniqueName: \"kubernetes.io/projected/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-kube-api-access-ggkk5\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:36:47.080604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.080600 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:36:47.080604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.080609 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:36:47.650478 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.650443 2567 generic.go:356] "Generic (PLEG): container finished" podID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" containerID="199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b" exitCode=0 Jul 14 18:36:47.650912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.650496 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" Jul 14 18:36:47.650912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.650534 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" event={"ID":"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7","Type":"ContainerDied","Data":"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b"} Jul 14 18:36:47.650912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.650622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx" event={"ID":"5ef93f3b-6b7d-4682-895e-bb18c7bd33f7","Type":"ContainerDied","Data":"480cdf9c8401047369a9c61180b1dbf0b4ec6e842b82f84015628cc57d6889e8"} Jul 14 18:36:47.650912 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.650649 2567 scope.go:117] "RemoveContainer" containerID="199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b" Jul 14 18:36:47.660922 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.660702 2567 scope.go:117] "RemoveContainer" containerID="199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b" Jul 14 18:36:47.661033 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:36:47.661012 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b\": container with ID starting with 199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b not found: ID does not exist" containerID="199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b" Jul 14 18:36:47.661090 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.661043 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b"} err="failed to get container status \"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b\": rpc error: code = NotFound desc = could not find container \"199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b\": container with ID starting with 199a6c254f1843262608f141f42670ae256cc8b479201c24773589764edbf57b not found: ID does not exist" Jul 14 18:36:47.673749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.673721 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:36:47.677725 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.677700 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-499cb51a-db8c5459-wb9bx"] Jul 14 18:36:47.931549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:47.931469 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" path="/var/lib/kubelet/pods/5ef93f3b-6b7d-4682-895e-bb18c7bd33f7/volumes" Jul 14 18:36:53.947691 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:53.947655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.048345 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.048310 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.371014 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.370959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.448117 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.448084 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.449004 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.448977 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:36:54.449592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.449562 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" containerName="maas-api" Jul 14 18:36:54.449592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.449582 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" containerName="maas-api" Jul 14 18:36:54.449750 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.449665 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef93f3b-6b7d-4682-895e-bb18c7bd33f7" containerName="maas-api" Jul 14 18:36:54.453898 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.453879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.456010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.455988 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1808df45-serving-cert\"" Jul 14 18:36:54.465593 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.465572 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:36:54.529774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.529741 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.550924 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.550891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.551085 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.550960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.551085 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.551068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lzp\" (UniqueName: \"kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.652212 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.652126 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lzp\" (UniqueName: \"kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.652212 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.652187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.652418 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.652225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.652852 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.652828 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.654712 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.654688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.662189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.662165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lzp\" (UniqueName: \"kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp\") pod \"maas-api-e2e-ait-1808df45-7d956696c6-r4fvw\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.675825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.675779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:54.765162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.765126 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:54.913747 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:54.913719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:36:54.916033 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:36:54.916005 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175dc5b7_ded2_4409_9ec0_ff35871183b5.slice/crio-ee0d18ee919a05d21c792140282deee7e91c4f159bce08a4b46cf50068be229d WatchSource:0}: Error finding container ee0d18ee919a05d21c792140282deee7e91c4f159bce08a4b46cf50068be229d: Status 404 returned error can't find the container with id ee0d18ee919a05d21c792140282deee7e91c4f159bce08a4b46cf50068be229d Jul 14 18:36:55.024781 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.024734 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:55.126908 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.126871 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:55.690222 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.690114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" event={"ID":"175dc5b7-ded2-4409-9ec0-ff35871183b5","Type":"ContainerStarted","Data":"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c"} Jul 14 18:36:55.690384 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.690315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:36:55.690384 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.690349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" event={"ID":"175dc5b7-ded2-4409-9ec0-ff35871183b5","Type":"ContainerStarted","Data":"ee0d18ee919a05d21c792140282deee7e91c4f159bce08a4b46cf50068be229d"} Jul 14 18:36:55.711017 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:55.710963 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" podStartSLOduration=1.317621472 podStartE2EDuration="1.710947498s" totalImagesPullingTime="393.326026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:36:55.708678504 +0000 UTC m=+1581.094576960" watchObservedRunningTime="2026-07-14 18:36:55.710947498 +0000 UTC m=+1581.096845951" Jul 14 18:36:56.719671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:56.719640 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:57.763027 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:57.762975 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:57.863383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:36:57.863343 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:01.702214 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:01.702181 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:37:10.088774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.088723 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:37:10.089285 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.088980 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" podUID="175dc5b7-ded2-4409-9ec0-ff35871183b5" containerName="maas-api" containerID="cri-o://e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c" gracePeriod=30 Jul 14 18:37:10.335591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.335561 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:37:10.411661 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.411563 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca\") pod \"175dc5b7-ded2-4409-9ec0-ff35871183b5\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " Jul 14 18:37:10.411661 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.411641 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44lzp\" (UniqueName: \"kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp\") pod \"175dc5b7-ded2-4409-9ec0-ff35871183b5\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " Jul 14 18:37:10.411949 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.411676 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls\") pod \"175dc5b7-ded2-4409-9ec0-ff35871183b5\" (UID: \"175dc5b7-ded2-4409-9ec0-ff35871183b5\") " Jul 14 18:37:10.412072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.412046 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "175dc5b7-ded2-4409-9ec0-ff35871183b5" (UID: "175dc5b7-ded2-4409-9ec0-ff35871183b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:37:10.413981 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.413953 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp" (OuterVolumeSpecName: "kube-api-access-44lzp") pod "175dc5b7-ded2-4409-9ec0-ff35871183b5" (UID: "175dc5b7-ded2-4409-9ec0-ff35871183b5"). InnerVolumeSpecName "kube-api-access-44lzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:37:10.414078 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.413997 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "175dc5b7-ded2-4409-9ec0-ff35871183b5" (UID: "175dc5b7-ded2-4409-9ec0-ff35871183b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:37:10.513335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.513300 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/175dc5b7-ded2-4409-9ec0-ff35871183b5-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:37:10.513335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.513329 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lzp\" (UniqueName: \"kubernetes.io/projected/175dc5b7-ded2-4409-9ec0-ff35871183b5-kube-api-access-44lzp\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:37:10.513335 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.513339 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/175dc5b7-ded2-4409-9ec0-ff35871183b5-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:37:10.766956 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.766920 2567 generic.go:356] "Generic (PLEG): container finished" podID="175dc5b7-ded2-4409-9ec0-ff35871183b5" containerID="e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c" exitCode=0 Jul 14 18:37:10.767127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.766975 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" Jul 14 18:37:10.767127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.766998 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" event={"ID":"175dc5b7-ded2-4409-9ec0-ff35871183b5","Type":"ContainerDied","Data":"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c"} Jul 14 18:37:10.767127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.767055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw" event={"ID":"175dc5b7-ded2-4409-9ec0-ff35871183b5","Type":"ContainerDied","Data":"ee0d18ee919a05d21c792140282deee7e91c4f159bce08a4b46cf50068be229d"} Jul 14 18:37:10.767127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.767078 2567 scope.go:117] "RemoveContainer" containerID="e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c" Jul 14 18:37:10.776844 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.776824 2567 scope.go:117] "RemoveContainer" containerID="e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c" Jul 14 18:37:10.777148 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:37:10.777131 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c\": container with ID starting with e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c not found: ID does not exist" containerID="e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c" Jul 14 18:37:10.777192 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.777157 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c"} err="failed to get container status \"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c\": rpc error: code = NotFound desc = could not find container \"e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c\": container with ID starting with e492b0d7d701888311b446f787b16ec36b90249fe88b74f873958172a5dfd85c not found: ID does not exist" Jul 14 18:37:10.790890 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.790863 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:37:10.798861 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.798791 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1808df45-7d956696c6-r4fvw"] Jul 14 18:37:10.899460 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:10.899424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:11.931229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:11.931195 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175dc5b7-ded2-4409-9ec0-ff35871183b5" path="/var/lib/kubelet/pods/175dc5b7-ded2-4409-9ec0-ff35871183b5/volumes" Jul 14 18:37:14.674996 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:14.674959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:21.388128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:21.388092 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:21.486933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:21.486902 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:25.758083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:25.758045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:26.089798 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:26.089760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:27.307415 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.307376 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:37:27.307980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.307956 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175dc5b7-ded2-4409-9ec0-ff35871183b5" containerName="maas-api" Jul 14 18:37:27.307980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.307982 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="175dc5b7-ded2-4409-9ec0-ff35871183b5" containerName="maas-api" Jul 14 18:37:27.308147 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.308100 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="175dc5b7-ded2-4409-9ec0-ff35871183b5" containerName="maas-api" Jul 14 18:37:27.312580 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.312562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.314530 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.314510 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e2e80b9d-serving-cert\"" Jul 14 18:37:27.322452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.322402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:27.323679 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.323655 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:37:27.383971 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.383931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.384133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.383976 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.384133 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.384042 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfrx\" (UniqueName: \"kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.416919 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.416884 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:27.484898 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.484862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.485102 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.484908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.485102 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.484945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfrx\" (UniqueName: \"kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.485519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.485496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.487176 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.487147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.494907 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.494883 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfrx\" (UniqueName: \"kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx\") pod \"maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.522035 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.521997 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:27.624314 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.624216 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:27.778707 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.778535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:37:27.781189 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:37:27.781151 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1e4a14_88d9_4d53_b1ba_43a4d6629636.slice/crio-cae4f2ab49e93e8f6b360029e32e741ecce9f9341207939911a58f772c122812 WatchSource:0}: Error finding container cae4f2ab49e93e8f6b360029e32e741ecce9f9341207939911a58f772c122812: Status 404 returned error can't find the container with id cae4f2ab49e93e8f6b360029e32e741ecce9f9341207939911a58f772c122812 Jul 14 18:37:27.859882 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:27.859838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" event={"ID":"cc1e4a14-88d9-4d53-b1ba-43a4d6629636","Type":"ContainerStarted","Data":"cae4f2ab49e93e8f6b360029e32e741ecce9f9341207939911a58f772c122812"} Jul 14 18:37:28.031528 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:28.031492 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:28.131716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:28.131684 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:28.868921 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:28.868877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" event={"ID":"cc1e4a14-88d9-4d53-b1ba-43a4d6629636","Type":"ContainerStarted","Data":"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41"} Jul 14 18:37:28.869305 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:28.869032 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:28.887980 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:28.887921 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" podStartSLOduration=1.525774732 podStartE2EDuration="1.887899653s" totalImagesPullingTime="362.124921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:37:28.887225296 +0000 UTC m=+1614.273123766" watchObservedRunningTime="2026-07-14 18:37:28.887899653 +0000 UTC m=+1614.273798107" Jul 14 18:37:32.266779 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:32.266740 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:32.337550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:32.337516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:37:34.878867 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:34.878832 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:37:35.238987 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:37:35.238956 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:17.575002 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:17.574967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:17.672637 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:17.672608 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:32.528749 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:32.528697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:32.628573 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:32.628533 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:42.258393 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.258358 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:38:42.258864 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.258619 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" podUID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" containerName="maas-api" containerID="cri-o://60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41" gracePeriod=30 Jul 14 18:38:42.279739 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.279714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:42.502452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.502422 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:38:42.625073 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.624986 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdfrx\" (UniqueName: \"kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx\") pod \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " Jul 14 18:38:42.625073 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.625037 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls\") pod \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " Jul 14 18:38:42.625270 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.625091 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca\") pod \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\" (UID: \"cc1e4a14-88d9-4d53-b1ba-43a4d6629636\") " Jul 14 18:38:42.625494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.625469 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc1e4a14-88d9-4d53-b1ba-43a4d6629636" (UID: "cc1e4a14-88d9-4d53-b1ba-43a4d6629636"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:38:42.627127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.627107 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx" (OuterVolumeSpecName: "kube-api-access-zdfrx") pod "cc1e4a14-88d9-4d53-b1ba-43a4d6629636" (UID: "cc1e4a14-88d9-4d53-b1ba-43a4d6629636"). InnerVolumeSpecName "kube-api-access-zdfrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:38:42.627301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.627274 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc1e4a14-88d9-4d53-b1ba-43a4d6629636" (UID: "cc1e4a14-88d9-4d53-b1ba-43a4d6629636"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:38:42.725915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.725880 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:38:42.725915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.725910 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdfrx\" (UniqueName: \"kubernetes.io/projected/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-kube-api-access-zdfrx\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:38:42.725915 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:42.725920 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc1e4a14-88d9-4d53-b1ba-43a4d6629636-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:38:43.251943 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.251903 2567 generic.go:356] "Generic (PLEG): container finished" podID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" containerID="60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41" exitCode=0 Jul 14 18:38:43.251943 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.251943 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" Jul 14 18:38:43.252151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.251954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" event={"ID":"cc1e4a14-88d9-4d53-b1ba-43a4d6629636","Type":"ContainerDied","Data":"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41"} Jul 14 18:38:43.252151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.252004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b" event={"ID":"cc1e4a14-88d9-4d53-b1ba-43a4d6629636","Type":"ContainerDied","Data":"cae4f2ab49e93e8f6b360029e32e741ecce9f9341207939911a58f772c122812"} Jul 14 18:38:43.252151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.252027 2567 scope.go:117] "RemoveContainer" containerID="60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41" Jul 14 18:38:43.265384 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.265177 2567 scope.go:117] "RemoveContainer" containerID="60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41" Jul 14 18:38:43.265694 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:38:43.265457 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41\": container with ID starting with 60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41 not found: ID does not exist" containerID="60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41" Jul 14 18:38:43.265694 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.265486 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41"} err="failed to get container status \"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41\": rpc error: code = NotFound desc = could not find container \"60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41\": container with ID starting with 60263d02b52a86f310a9b47da7b03a25a8914362372604f6d6df20a9c4393f41 not found: ID does not exist" Jul 14 18:38:43.275353 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.275328 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:38:43.279751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.279726 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2e80b9d-7fcfff6cc6-74x8b"] Jul 14 18:38:43.931162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:43.931132 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" path="/var/lib/kubelet/pods/cc1e4a14-88d9-4d53-b1ba-43a4d6629636/volumes" Jul 14 18:38:46.849380 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:46.849341 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:53.682278 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:53.682241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:53.782653 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:53.782620 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:57.075208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:57.075121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:57.793267 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:57.793234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:57.893604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:57.893569 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:57.993356 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:57.993320 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:59.573449 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:38:59.573408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:03.575014 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:03.574982 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:05.933216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:05.933180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:06.036381 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:06.036351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:06.172623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:06.172593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:06.272608 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:06.272571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:06.729148 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:06.729102 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:32.790164 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:32.790129 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:39.297908 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:39.297876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:43.895077 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:43.895007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:39:55.524034 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:39:55.524000 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:02.110017 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:02.109982 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:02.218029 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:02.217988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:02.409955 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:02.409875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:12.927958 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:12.927922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:13.852348 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:13.852297 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:20.316876 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:20.316840 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:20.373421 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:20.373386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:20.470567 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:20.470534 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:20.564868 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:20.564833 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:20.665691 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:20.665605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:21.542978 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:21.542948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:21.649788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:21.649761 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:21.755486 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:21.755453 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:23.613730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:23.613679 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:24.282328 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:24.282291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:30.276906 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:30.276797 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:30.342630 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:30.342598 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:31.000987 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:31.000952 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:36.079029 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:36.078916 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:40:36.087514 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:36.079248 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:40:37.279684 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:37.279646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:37.379492 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:37.379451 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:37.540700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:37.540594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:37.873502 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:37.873410 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:38.067615 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:38.067562 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:41.116397 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.116353 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:41.214922 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.214885 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:41.369713 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.369638 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:40:41.370209 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.370192 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" containerName="maas-api" Jul 14 18:40:41.370255 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.370214 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" containerName="maas-api" Jul 14 18:40:41.370298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.370287 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1e4a14-88d9-4d53-b1ba-43a4d6629636" containerName="maas-api" Jul 14 18:40:41.373381 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.373359 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.375596 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.375575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-467eca84-serving-cert\"" Jul 14 18:40:41.383633 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.383606 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:40:41.397858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.397829 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:41.472968 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.472938 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.473138 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.472998 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.473138 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.473053 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.498745 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.498715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:41.573970 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.573932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.574144 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.574005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.574144 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.574055 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.574742 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.574714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.576468 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.576448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.584385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.584358 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv\") pod \"maas-api-e2e-mt-467eca84-5b445cd857-96qqh\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.685047 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.684952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:41.928243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:41.928207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:42.029196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.029149 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:42.030934 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.030915 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:40:42.033231 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:40:42.033205 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529e03a_99dd_40e0_803d_f123a764fbc4.slice/crio-3a5832b3ee3610c701b2838966cf9334baafe00ac74989189b175f7a15cc6faa WatchSource:0}: Error finding container 3a5832b3ee3610c701b2838966cf9334baafe00ac74989189b175f7a15cc6faa: Status 404 returned error can't find the container with id 3a5832b3ee3610c701b2838966cf9334baafe00ac74989189b175f7a15cc6faa Jul 14 18:40:42.034481 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.034464 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:40:42.850317 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.850209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" event={"ID":"3529e03a-99dd-40e0-803d-f123a764fbc4","Type":"ContainerStarted","Data":"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b"} Jul 14 18:40:42.850317 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.850271 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" event={"ID":"3529e03a-99dd-40e0-803d-f123a764fbc4","Type":"ContainerStarted","Data":"3a5832b3ee3610c701b2838966cf9334baafe00ac74989189b175f7a15cc6faa"} Jul 14 18:40:42.850317 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.850312 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:42.869654 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:42.869597 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" podStartSLOduration=1.317087038 podStartE2EDuration="1.869582202s" totalImagesPullingTime="552.495164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:40:42.868061901 +0000 UTC m=+1808.253960367" watchObservedRunningTime="2026-07-14 18:40:42.869582202 +0000 UTC m=+1808.255480711" Jul 14 18:40:45.675736 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:45.675684 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:46.404428 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:46.404395 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:46.502659 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:46.502620 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:46.906605 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:46.906570 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:48.861199 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:48.861168 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:40:53.474558 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:53.474527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:53.578201 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:40:53.578167 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:04.388172 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.388134 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:41:04.388649 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.388403 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" podUID="3529e03a-99dd-40e0-803d-f123a764fbc4" containerName="maas-api" containerID="cri-o://10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b" gracePeriod=30 Jul 14 18:41:04.413445 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.413413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:04.639493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.639434 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:41:04.703120 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.703082 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls\") pod \"3529e03a-99dd-40e0-803d-f123a764fbc4\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " Jul 14 18:41:04.703292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.703272 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv\") pod \"3529e03a-99dd-40e0-803d-f123a764fbc4\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " Jul 14 18:41:04.703338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.703326 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca\") pod \"3529e03a-99dd-40e0-803d-f123a764fbc4\" (UID: \"3529e03a-99dd-40e0-803d-f123a764fbc4\") " Jul 14 18:41:04.703790 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.703762 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3529e03a-99dd-40e0-803d-f123a764fbc4" (UID: "3529e03a-99dd-40e0-803d-f123a764fbc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:41:04.705254 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.705229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3529e03a-99dd-40e0-803d-f123a764fbc4" (UID: "3529e03a-99dd-40e0-803d-f123a764fbc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:41:04.705417 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.705398 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv" (OuterVolumeSpecName: "kube-api-access-pstsv") pod "3529e03a-99dd-40e0-803d-f123a764fbc4" (UID: "3529e03a-99dd-40e0-803d-f123a764fbc4"). InnerVolumeSpecName "kube-api-access-pstsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:41:04.804260 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.804220 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3529e03a-99dd-40e0-803d-f123a764fbc4-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:41:04.804260 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.804256 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstsv\" (UniqueName: \"kubernetes.io/projected/3529e03a-99dd-40e0-803d-f123a764fbc4-kube-api-access-pstsv\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:41:04.804500 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.804273 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3529e03a-99dd-40e0-803d-f123a764fbc4-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:41:04.963552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.963458 2567 generic.go:356] "Generic (PLEG): container finished" podID="3529e03a-99dd-40e0-803d-f123a764fbc4" containerID="10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b" exitCode=0 Jul 14 18:41:04.963552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.963509 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" Jul 14 18:41:04.963552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.963528 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" event={"ID":"3529e03a-99dd-40e0-803d-f123a764fbc4","Type":"ContainerDied","Data":"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b"} Jul 14 18:41:04.963857 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.963567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh" event={"ID":"3529e03a-99dd-40e0-803d-f123a764fbc4","Type":"ContainerDied","Data":"3a5832b3ee3610c701b2838966cf9334baafe00ac74989189b175f7a15cc6faa"} Jul 14 18:41:04.963857 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.963589 2567 scope.go:117] "RemoveContainer" containerID="10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b" Jul 14 18:41:04.973238 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.973213 2567 scope.go:117] "RemoveContainer" containerID="10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b" Jul 14 18:41:04.973526 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:41:04.973504 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b\": container with ID starting with 10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b not found: ID does not exist" containerID="10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b" Jul 14 18:41:04.973570 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.973536 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b"} err="failed to get container status \"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b\": rpc error: code = NotFound desc = could not find container \"10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b\": container with ID starting with 10916663d6f0a2fb7ad13d0a7a35dba58fcec7f0272b9a110f083bd9779c3c7b not found: ID does not exist" Jul 14 18:41:04.987596 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.987568 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:41:04.993039 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:04.993015 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-467eca84-5b445cd857-96qqh"] Jul 14 18:41:05.931167 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:05.931136 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3529e03a-99dd-40e0-803d-f123a764fbc4" path="/var/lib/kubelet/pods/3529e03a-99dd-40e0-803d-f123a764fbc4/volumes" Jul 14 18:41:08.977144 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:08.977112 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:15.852030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:15.851996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:15.914956 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:15.914915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:19.776979 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:19.776941 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:20.044928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:20.044837 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:20.146338 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:20.146293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:20.248428 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:20.248400 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:20.401465 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:20.401379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:26.870399 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:26.870361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:26.978443 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:26.978412 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:27.072207 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:27.072159 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:27.247797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:27.247757 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:33.805717 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:33.805682 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:33.906858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:33.906794 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:40.844591 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:40.844552 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:41.176559 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:41.176466 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:41.670339 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:41.670310 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:44.430112 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.430064 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:44.527734 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.527590 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:41:44.528260 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.528231 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3529e03a-99dd-40e0-803d-f123a764fbc4" containerName="maas-api" Jul 14 18:41:44.528260 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.528261 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3529e03a-99dd-40e0-803d-f123a764fbc4" containerName="maas-api" Jul 14 18:41:44.528452 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.528368 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3529e03a-99dd-40e0-803d-f123a764fbc4" containerName="maas-api" Jul 14 18:41:44.528703 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.528683 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:44.533001 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.532983 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.535459 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.535439 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c99ca1c1-serving-cert\"" Jul 14 18:41:44.544343 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.544318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:41:44.588466 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.588433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.588658 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.588470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw6w\" (UniqueName: \"kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.588658 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.588560 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.690115 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.690020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.690115 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.690060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw6w\" (UniqueName: \"kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.690303 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.690194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.690341 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:41:44.690326 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-serving-cert: secret "maas-api-e2e-mt-c99ca1c1-serving-cert" not found Jul 14 18:41:44.690429 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:41:44.690419 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls podName:a75b80fe-3ee5-4627-9926-39fb7cc392db nodeName:}" failed. No retries permitted until 2026-07-14 18:41:45.190402295 +0000 UTC m=+1870.576300726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls") pod "maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" (UID: "a75b80fe-3ee5-4627-9926-39fb7cc392db") : secret "maas-api-e2e-mt-c99ca1c1-serving-cert" not found Jul 14 18:41:44.690721 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.690703 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:44.702668 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:44.702637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw6w\" (UniqueName: \"kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:45.196278 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:45.196234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:45.198918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:45.198892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") pod \"maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:45.444802 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:45.444764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:45.590303 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:45.590275 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:41:45.592497 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:41:45.592466 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75b80fe_3ee5_4627_9926_39fb7cc392db.slice/crio-3ba6c73455f80ba5b40efe6d2667936e35c5b0bc566ad658a412cb5c71fd7097 WatchSource:0}: Error finding container 3ba6c73455f80ba5b40efe6d2667936e35c5b0bc566ad658a412cb5c71fd7097: Status 404 returned error can't find the container with id 3ba6c73455f80ba5b40efe6d2667936e35c5b0bc566ad658a412cb5c71fd7097 Jul 14 18:41:46.171124 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:46.171087 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" event={"ID":"a75b80fe-3ee5-4627-9926-39fb7cc392db","Type":"ContainerStarted","Data":"095033235c3c36bb68d972f2183e3c4a1de9701cada9ed139e2f818d79c441a3"} Jul 14 18:41:46.171302 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:46.171141 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" event={"ID":"a75b80fe-3ee5-4627-9926-39fb7cc392db","Type":"ContainerStarted","Data":"3ba6c73455f80ba5b40efe6d2667936e35c5b0bc566ad658a412cb5c71fd7097"} Jul 14 18:41:46.171302 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:46.171204 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:46.192623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:46.192562 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" podStartSLOduration=1.7984354150000001 podStartE2EDuration="2.192541861s" totalImagesPullingTime="394.106446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:41:46.191103352 +0000 UTC m=+1871.577001803" watchObservedRunningTime="2026-07-14 18:41:46.192541861 +0000 UTC m=+1871.578440310" Jul 14 18:41:47.561859 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:47.561804 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:47.621700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:47.621664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:48.948560 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:48.948525 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:52.181797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:52.181766 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:41:57.376169 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:41:57.376089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:42:12.091001 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:12.090968 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:42:12.195711 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:12.195675 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:42:38.200889 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.200851 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:42:38.201796 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.201740 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" podUID="a75b80fe-3ee5-4627-9926-39fb7cc392db" containerName="maas-api" containerID="cri-o://095033235c3c36bb68d972f2183e3c4a1de9701cada9ed139e2f818d79c441a3" gracePeriod=30 Jul 14 18:42:38.237257 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.237217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:42:38.429694 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.429652 2567 generic.go:356] "Generic (PLEG): container finished" podID="a75b80fe-3ee5-4627-9926-39fb7cc392db" containerID="095033235c3c36bb68d972f2183e3c4a1de9701cada9ed139e2f818d79c441a3" exitCode=0 Jul 14 18:42:38.429877 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.429722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" event={"ID":"a75b80fe-3ee5-4627-9926-39fb7cc392db","Type":"ContainerDied","Data":"095033235c3c36bb68d972f2183e3c4a1de9701cada9ed139e2f818d79c441a3"} Jul 14 18:42:38.473585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.473553 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:42:38.618208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.618154 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlw6w\" (UniqueName: \"kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w\") pod \"a75b80fe-3ee5-4627-9926-39fb7cc392db\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " Jul 14 18:42:38.618417 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.618242 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") pod \"a75b80fe-3ee5-4627-9926-39fb7cc392db\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " Jul 14 18:42:38.618417 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.618278 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca\") pod \"a75b80fe-3ee5-4627-9926-39fb7cc392db\" (UID: \"a75b80fe-3ee5-4627-9926-39fb7cc392db\") " Jul 14 18:42:38.618662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.618635 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a75b80fe-3ee5-4627-9926-39fb7cc392db" (UID: "a75b80fe-3ee5-4627-9926-39fb7cc392db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:42:38.620377 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.620348 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a75b80fe-3ee5-4627-9926-39fb7cc392db" (UID: "a75b80fe-3ee5-4627-9926-39fb7cc392db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:42:38.620484 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.620352 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w" (OuterVolumeSpecName: "kube-api-access-hlw6w") pod "a75b80fe-3ee5-4627-9926-39fb7cc392db" (UID: "a75b80fe-3ee5-4627-9926-39fb7cc392db"). InnerVolumeSpecName "kube-api-access-hlw6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:42:38.720198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.720098 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlw6w\" (UniqueName: \"kubernetes.io/projected/a75b80fe-3ee5-4627-9926-39fb7cc392db-kube-api-access-hlw6w\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:42:38.720198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.720142 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75b80fe-3ee5-4627-9926-39fb7cc392db-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:42:38.720198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:38.720152 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75b80fe-3ee5-4627-9926-39fb7cc392db-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:42:39.435938 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.435906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" event={"ID":"a75b80fe-3ee5-4627-9926-39fb7cc392db","Type":"ContainerDied","Data":"3ba6c73455f80ba5b40efe6d2667936e35c5b0bc566ad658a412cb5c71fd7097"} Jul 14 18:42:39.436324 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.435965 2567 scope.go:117] "RemoveContainer" containerID="095033235c3c36bb68d972f2183e3c4a1de9701cada9ed139e2f818d79c441a3" Jul 14 18:42:39.436324 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.435917 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9" Jul 14 18:42:39.460137 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.460090 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:42:39.462641 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.462617 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99ca1c1-56c46c59c-trcv9"] Jul 14 18:42:39.931227 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:39.931196 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75b80fe-3ee5-4627-9926-39fb7cc392db" path="/var/lib/kubelet/pods/a75b80fe-3ee5-4627-9926-39fb7cc392db/volumes" Jul 14 18:42:42.596229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:42.596195 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:42:47.763074 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:42:47.763041 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:18.445093 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:18.445014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:18.543046 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:18.542973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:24.988210 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:24.988180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:25.742618 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:25.742579 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:34.394954 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:34.394874 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:34.491803 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:34.491774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:35.265369 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:35.265334 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:35.524096 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:35.524008 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:36.798518 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:36.798488 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:36.898640 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:36.898608 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:36.996879 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:36.996851 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:37.902933 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:37.902902 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:37.997604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:37.997572 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:38.097466 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:38.097426 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:39.725313 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:39.725278 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:46.409552 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:46.409516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:46.511665 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:46.511629 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:47.174026 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:47.173995 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:55.171157 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:55.171121 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:55.264250 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:55.264211 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:55.365565 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:55.365530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:43:55.469443 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:43:55.469359 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:06.023507 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:06.023470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:06.415916 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:06.415780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:27.741794 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:27.741760 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:35.186320 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:35.186288 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:36.680062 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:36.679999 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:36.781543 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:36.781492 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:39.401132 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.401091 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:39.486932 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.486890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:39.488613 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.488587 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:44:39.489038 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.489024 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75b80fe-3ee5-4627-9926-39fb7cc392db" containerName="maas-api" Jul 14 18:44:39.489093 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.489039 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75b80fe-3ee5-4627-9926-39fb7cc392db" containerName="maas-api" Jul 14 18:44:39.489128 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.489124 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75b80fe-3ee5-4627-9926-39fb7cc392db" containerName="maas-api" Jul 14 18:44:39.492101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.492084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.494154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.494136 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f114d-serving-cert\"" Jul 14 18:44:39.495518 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.495495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldjd\" (UniqueName: \"kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.495613 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.495557 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.495672 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.495605 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.502674 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.502651 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:44:39.566652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.566602 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:39.596274 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.596239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldjd\" (UniqueName: \"kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.596434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.596382 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.596434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.596417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.596572 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:44:39.596558 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-serving-cert: secret "maas-api-e2e-inf-a-1f114d-serving-cert" not found Jul 14 18:44:39.596636 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:44:39.596627 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls podName:983d855b-5ae5-45ca-a93f-af2f1b6e6dda nodeName:}" failed. No retries permitted until 2026-07-14 18:44:40.096610263 +0000 UTC m=+2045.482508694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls") pod "maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" (UID: "983d855b-5ae5-45ca-a93f-af2f1b6e6dda") : secret "maas-api-e2e-inf-a-1f114d-serving-cert" not found Jul 14 18:44:39.596986 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.596970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.611474 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.611445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldjd\" (UniqueName: \"kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:39.723479 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:39.723402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:40.101162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.101119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:40.103888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.103863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f114d-58f497649b-8zh28\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:40.124507 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.124480 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:40.225494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.225454 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:40.403249 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.403155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:40.549559 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:40.549507 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:44:40.551956 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:44:40.551923 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983d855b_5ae5_45ca_a93f_af2f1b6e6dda.slice/crio-e8ed5082b46124b9ac812fa939d303f854b45e320ab17356d6a368b9374f33da WatchSource:0}: Error finding container e8ed5082b46124b9ac812fa939d303f854b45e320ab17356d6a368b9374f33da: Status 404 returned error can't find the container with id e8ed5082b46124b9ac812fa939d303f854b45e320ab17356d6a368b9374f33da Jul 14 18:44:41.046270 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:41.046234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" event={"ID":"983d855b-5ae5-45ca-a93f-af2f1b6e6dda","Type":"ContainerStarted","Data":"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df"} Jul 14 18:44:41.046468 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:41.046283 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" event={"ID":"983d855b-5ae5-45ca-a93f-af2f1b6e6dda","Type":"ContainerStarted","Data":"e8ed5082b46124b9ac812fa939d303f854b45e320ab17356d6a368b9374f33da"} Jul 14 18:44:41.046468 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:41.046305 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:41.066702 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:41.066653 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" podStartSLOduration=1.662427949 podStartE2EDuration="2.066637432s" totalImagesPullingTime="404.209483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:44:41.065198586 +0000 UTC m=+2046.451097042" watchObservedRunningTime="2026-07-14 18:44:41.066637432 +0000 UTC m=+2046.452535884" Jul 14 18:44:43.986268 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:43.986230 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:45.411141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:45.411107 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:47.056769 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:47.056741 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:44:47.917896 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:47.917858 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:47.994362 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:47.994326 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:44:47.995088 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:47.995052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:47.998860 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:47.998839 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.002970 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.002953 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7965f3-serving-cert\"" Jul 14 18:44:48.010196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.010170 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:44:48.071494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.071454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.071494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.071500 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.071991 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.071581 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jcc\" (UniqueName: \"kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.074684 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.074657 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:48.172471 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.172376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jcc\" (UniqueName: \"kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.172652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.172484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.172652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.172511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.172652 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:44:48.172640 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-serving-cert: secret "maas-api-e2e-inf-b-7965f3-serving-cert" not found Jul 14 18:44:48.172840 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:44:48.172718 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls podName:27ccb0c7-4e59-436b-a0f9-0a568423049e nodeName:}" failed. No retries permitted until 2026-07-14 18:44:48.672697153 +0000 UTC m=+2054.058595589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls") pod "maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" (UID: "27ccb0c7-4e59-436b-a0f9-0a568423049e") : secret "maas-api-e2e-inf-b-7965f3-serving-cert" not found Jul 14 18:44:48.173152 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.173134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.183224 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.183196 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jcc\" (UniqueName: \"kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.411998 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.411950 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:48.677740 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.677706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.680189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.680155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") pod \"maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:48.717867 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.717801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:48.818249 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.818215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:48.911326 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:48.911286 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:49.053759 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:49.053730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:44:49.055779 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:44:49.055751 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ccb0c7_4e59_436b_a0f9_0a568423049e.slice/crio-96527e33f38a94f487092e8433970bd430749a0cd6063dfb6cbe4642e584fc10 WatchSource:0}: Error finding container 96527e33f38a94f487092e8433970bd430749a0cd6063dfb6cbe4642e584fc10: Status 404 returned error can't find the container with id 96527e33f38a94f487092e8433970bd430749a0cd6063dfb6cbe4642e584fc10 Jul 14 18:44:49.094016 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:49.093982 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" event={"ID":"27ccb0c7-4e59-436b-a0f9-0a568423049e","Type":"ContainerStarted","Data":"96527e33f38a94f487092e8433970bd430749a0cd6063dfb6cbe4642e584fc10"} Jul 14 18:44:50.100259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:50.100223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" event={"ID":"27ccb0c7-4e59-436b-a0f9-0a568423049e","Type":"ContainerStarted","Data":"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931"} Jul 14 18:44:50.100647 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:50.100279 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:44:50.121145 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:50.121096 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" podStartSLOduration=2.7328863979999998 podStartE2EDuration="3.121075864s" totalImagesPullingTime="388.189466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:44:50.120203614 +0000 UTC m=+2055.506102066" watchObservedRunningTime="2026-07-14 18:44:50.121075864 +0000 UTC m=+2055.506974320" Jul 14 18:44:52.479191 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:52.479155 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:44:56.110836 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:44:56.110786 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:45:01.436229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:01.436191 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:01.496549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:01.496514 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:05.094561 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:05.094524 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:10.262883 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:10.262798 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:10.363519 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:10.363481 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:16.698010 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:16.697975 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:16.797149 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:16.797107 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:36.113759 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:36.113656 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:45:36.116510 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:36.116487 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:45:38.096955 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:45:38.096888 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:12.073779 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:12.073742 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:13.046400 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:13.046364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:13.780608 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:13.780576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:13.880860 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:13.880822 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:23.749545 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:23.749510 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:46:23.750063 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:23.749788 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" podUID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" containerName="maas-api" containerID="cri-o://cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df" gracePeriod=30 Jul 14 18:46:23.773593 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:23.773567 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:23.996141 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:23.996117 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:46:24.120800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.120694 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ldjd\" (UniqueName: \"kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd\") pod \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " Jul 14 18:46:24.120800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.120762 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") pod \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " Jul 14 18:46:24.121070 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.120891 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca\") pod \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\" (UID: \"983d855b-5ae5-45ca-a93f-af2f1b6e6dda\") " Jul 14 18:46:24.121301 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.121271 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "983d855b-5ae5-45ca-a93f-af2f1b6e6dda" (UID: "983d855b-5ae5-45ca-a93f-af2f1b6e6dda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:46:24.122857 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.122828 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "983d855b-5ae5-45ca-a93f-af2f1b6e6dda" (UID: "983d855b-5ae5-45ca-a93f-af2f1b6e6dda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:46:24.122975 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.122951 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd" (OuterVolumeSpecName: "kube-api-access-2ldjd") pod "983d855b-5ae5-45ca-a93f-af2f1b6e6dda" (UID: "983d855b-5ae5-45ca-a93f-af2f1b6e6dda"). InnerVolumeSpecName "kube-api-access-2ldjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:24.221807 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.221767 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:24.221807 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.221802 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldjd\" (UniqueName: \"kubernetes.io/projected/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-kube-api-access-2ldjd\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:24.221807 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.221839 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/983d855b-5ae5-45ca-a93f-af2f1b6e6dda-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:24.584831 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.584778 2567 generic.go:356] "Generic (PLEG): container finished" podID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" containerID="cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df" exitCode=0 Jul 14 18:46:24.585030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.584858 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" Jul 14 18:46:24.585030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.584871 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" event={"ID":"983d855b-5ae5-45ca-a93f-af2f1b6e6dda","Type":"ContainerDied","Data":"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df"} Jul 14 18:46:24.585030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.584953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28" event={"ID":"983d855b-5ae5-45ca-a93f-af2f1b6e6dda","Type":"ContainerDied","Data":"e8ed5082b46124b9ac812fa939d303f854b45e320ab17356d6a368b9374f33da"} Jul 14 18:46:24.585030 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.584976 2567 scope.go:117] "RemoveContainer" containerID="cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df" Jul 14 18:46:24.598337 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.598318 2567 scope.go:117] "RemoveContainer" containerID="cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df" Jul 14 18:46:24.598643 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:46:24.598621 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df\": container with ID starting with cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df not found: ID does not exist" containerID="cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df" Jul 14 18:46:24.598733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.598652 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df"} err="failed to get container status \"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df\": rpc error: code = NotFound desc = could not find container \"cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df\": container with ID starting with cc888cf245fb34285d1b537de8e225d6951a0eed90dd84b278139e73b45587df not found: ID does not exist" Jul 14 18:46:24.607629 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.607604 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:46:24.612393 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:24.612371 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f114d-58f497649b-8zh28"] Jul 14 18:46:25.931354 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:25.931320 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" path="/var/lib/kubelet/pods/983d855b-5ae5-45ca-a93f-af2f1b6e6dda/volumes" Jul 14 18:46:28.313645 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:28.313571 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:35.149195 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:35.149160 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:35.249697 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:35.249665 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:39.044929 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:39.044890 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:39.299741 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:39.299648 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:40.179232 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:40.179199 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:40.277111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:40.277076 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:49.988252 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:49.988213 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:46:49.988700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:49.988427 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" podUID="27ccb0c7-4e59-436b-a0f9-0a568423049e" containerName="maas-api" containerID="cri-o://fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931" gracePeriod=30 Jul 14 18:46:50.022849 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.022791 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:46:50.238508 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.238448 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:46:50.366451 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.366410 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca\") pod \"27ccb0c7-4e59-436b-a0f9-0a568423049e\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " Jul 14 18:46:50.366655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.366485 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jcc\" (UniqueName: \"kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc\") pod \"27ccb0c7-4e59-436b-a0f9-0a568423049e\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " Jul 14 18:46:50.366655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.366547 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") pod \"27ccb0c7-4e59-436b-a0f9-0a568423049e\" (UID: \"27ccb0c7-4e59-436b-a0f9-0a568423049e\") " Jul 14 18:46:50.366805 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.366778 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27ccb0c7-4e59-436b-a0f9-0a568423049e" (UID: "27ccb0c7-4e59-436b-a0f9-0a568423049e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:46:50.368614 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.368590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27ccb0c7-4e59-436b-a0f9-0a568423049e" (UID: "27ccb0c7-4e59-436b-a0f9-0a568423049e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:46:50.368757 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.368632 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc" (OuterVolumeSpecName: "kube-api-access-j5jcc") pod "27ccb0c7-4e59-436b-a0f9-0a568423049e" (UID: "27ccb0c7-4e59-436b-a0f9-0a568423049e"). InnerVolumeSpecName "kube-api-access-j5jcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:46:50.467280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.467238 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jcc\" (UniqueName: \"kubernetes.io/projected/27ccb0c7-4e59-436b-a0f9-0a568423049e-kube-api-access-j5jcc\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:50.467280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.467274 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ccb0c7-4e59-436b-a0f9-0a568423049e-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:50.467280 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.467284 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ccb0c7-4e59-436b-a0f9-0a568423049e-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:46:50.718427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.718341 2567 generic.go:356] "Generic (PLEG): container finished" podID="27ccb0c7-4e59-436b-a0f9-0a568423049e" containerID="fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931" exitCode=0 Jul 14 18:46:50.718427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.718384 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" Jul 14 18:46:50.718614 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.718422 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" event={"ID":"27ccb0c7-4e59-436b-a0f9-0a568423049e","Type":"ContainerDied","Data":"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931"} Jul 14 18:46:50.718614 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.718472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m" event={"ID":"27ccb0c7-4e59-436b-a0f9-0a568423049e","Type":"ContainerDied","Data":"96527e33f38a94f487092e8433970bd430749a0cd6063dfb6cbe4642e584fc10"} Jul 14 18:46:50.718614 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.718493 2567 scope.go:117] "RemoveContainer" containerID="fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931" Jul 14 18:46:50.728062 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.728045 2567 scope.go:117] "RemoveContainer" containerID="fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931" Jul 14 18:46:50.728306 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:46:50.728286 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931\": container with ID starting with fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931 not found: ID does not exist" containerID="fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931" Jul 14 18:46:50.728399 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.728311 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931"} err="failed to get container status \"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931\": rpc error: code = NotFound desc = could not find container \"fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931\": container with ID starting with fe3994640eecddaa99f71573efb6896e09668121b8c25a54d8671c950422b931 not found: ID does not exist" Jul 14 18:46:50.741695 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.741665 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:46:50.747069 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:50.747044 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7965f3-c6c6f9944-6lp8m"] Jul 14 18:46:51.931495 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:51.931464 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ccb0c7-4e59-436b-a0f9-0a568423049e" path="/var/lib/kubelet/pods/27ccb0c7-4e59-436b-a0f9-0a568423049e/volumes" Jul 14 18:46:54.543959 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:46:54.543928 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:02.274797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:02.274765 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:02.339142 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:02.339105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:05.371297 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:05.371240 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:05.603475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:05.603438 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:05.998574 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:05.998544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:06.328797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:06.328697 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:06.446596 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:06.446566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.143948 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.143899 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.230410 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.230378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.231527 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.231502 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:09.231951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.231932 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" containerName="maas-api" Jul 14 18:47:09.231951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.231946 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" containerName="maas-api" Jul 14 18:47:09.232111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.231971 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ccb0c7-4e59-436b-a0f9-0a568423049e" containerName="maas-api" Jul 14 18:47:09.232111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.231977 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ccb0c7-4e59-436b-a0f9-0a568423049e" containerName="maas-api" Jul 14 18:47:09.232111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.232038 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="983d855b-5ae5-45ca-a93f-af2f1b6e6dda" containerName="maas-api" Jul 14 18:47:09.232111 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.232046 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ccb0c7-4e59-436b-a0f9-0a568423049e" containerName="maas-api" Jul 14 18:47:09.236672 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.236653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.238764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.238745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fade18-serving-cert\"" Jul 14 18:47:09.245804 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.245781 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:09.257381 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.257358 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.257506 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.257437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.257604 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.257588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4cm\" (UniqueName: \"kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.293176 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.293138 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.358475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.358435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.358475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.358477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.358732 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.358573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4cm\" (UniqueName: \"kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.358732 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:47:09.358589 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-serving-cert: secret "maas-api-e2e-api-a-fade18-serving-cert" not found Jul 14 18:47:09.358732 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:47:09.358679 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls podName:99aec02a-ce53-4b4b-b452-ededfa8b4603 nodeName:}" failed. No retries permitted until 2026-07-14 18:47:09.858662938 +0000 UTC m=+2195.244561373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls") pod "maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" (UID: "99aec02a-ce53-4b4b-b452-ededfa8b4603") : secret "maas-api-e2e-api-a-fade18-serving-cert" not found Jul 14 18:47:09.359234 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.359216 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.382237 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.382200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4cm\" (UniqueName: \"kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.418676 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.418575 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.847007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.846957 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:09.864118 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.864082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.866590 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.866565 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") pod \"maas-api-e2e-api-a-fade18-69fdd96979-cmsb7\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:09.947464 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:09.947415 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:10.147782 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:10.147689 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:10.285891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:10.285865 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:10.287638 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:47:10.287612 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99aec02a_ce53_4b4b_b452_ededfa8b4603.slice/crio-b59a028722576d5928b1efd248e5a4b3c0f81a4f398c19fdedcaabff5f9f0504 WatchSource:0}: Error finding container b59a028722576d5928b1efd248e5a4b3c0f81a4f398c19fdedcaabff5f9f0504: Status 404 returned error can't find the container with id b59a028722576d5928b1efd248e5a4b3c0f81a4f398c19fdedcaabff5f9f0504 Jul 14 18:47:10.289050 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:10.289031 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:47:10.825930 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:10.825895 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" event={"ID":"99aec02a-ce53-4b4b-b452-ededfa8b4603","Type":"ContainerStarted","Data":"b59a028722576d5928b1efd248e5a4b3c0f81a4f398c19fdedcaabff5f9f0504"} Jul 14 18:47:11.833184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:11.833144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" event={"ID":"99aec02a-ce53-4b4b-b452-ededfa8b4603","Type":"ContainerStarted","Data":"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f"} Jul 14 18:47:11.833546 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:11.833217 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:11.854392 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:11.854334 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" podStartSLOduration=2.225470472 podStartE2EDuration="2.854317153s" totalImagesPullingTime="628.846681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:47:11.85384604 +0000 UTC m=+2197.239744494" watchObservedRunningTime="2026-07-14 18:47:11.854317153 +0000 UTC m=+2197.240215606" Jul 14 18:47:13.690972 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:13.690939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:14.210424 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:14.210394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:14.539512 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:14.539471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:14.867687 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:14.867596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:17.722157 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.722117 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:17.792340 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.792304 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:47:17.795291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.795268 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:17.796022 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.796007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.798024 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.797991 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fde6bc-serving-cert\"" Jul 14 18:47:17.805662 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.805634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:47:17.841028 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.840990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.841212 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.841091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5sk\" (UniqueName: \"kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.841212 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.841158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.843230 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.843211 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:17.888507 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.888467 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:17.942208 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.942178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.942426 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.942401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.942549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.942528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5sk\" (UniqueName: \"kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.942549 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:47:17.942533 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-serving-cert: secret "maas-api-e2e-api-b-fde6bc-serving-cert" not found Jul 14 18:47:17.942672 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:47:17.942604 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls podName:c474453c-f466-4120-a059-a595d24d399f nodeName:}" failed. No retries permitted until 2026-07-14 18:47:18.442582288 +0000 UTC m=+2203.828480730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls") pod "maas-api-e2e-api-b-fde6bc-6948745577-s4622" (UID: "c474453c-f466-4120-a059-a595d24d399f") : secret "maas-api-e2e-api-b-fde6bc-serving-cert" not found Jul 14 18:47:17.942949 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.942931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:17.952132 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:17.952113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5sk\" (UniqueName: \"kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:18.145022 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.144986 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:18.447858 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.447729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:18.450253 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.450229 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") pod \"maas-api-e2e-api-b-fde6bc-6948745577-s4622\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:18.510805 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.510771 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:18.614729 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.614699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:18.708761 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.708652 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:18.854728 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.854704 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:47:18.856331 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:47:18.856288 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc474453c_f466_4120_a059_a595d24d399f.slice/crio-d7b3ff440fbf377a5711ea8dacfcfb9e1360d25d84da8f8e0fd0b560602c5032 WatchSource:0}: Error finding container d7b3ff440fbf377a5711ea8dacfcfb9e1360d25d84da8f8e0fd0b560602c5032: Status 404 returned error can't find the container with id d7b3ff440fbf377a5711ea8dacfcfb9e1360d25d84da8f8e0fd0b560602c5032 Jul 14 18:47:18.869055 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:18.869028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" event={"ID":"c474453c-f466-4120-a059-a595d24d399f","Type":"ContainerStarted","Data":"d7b3ff440fbf377a5711ea8dacfcfb9e1360d25d84da8f8e0fd0b560602c5032"} Jul 14 18:47:19.875770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:19.875725 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" event={"ID":"c474453c-f466-4120-a059-a595d24d399f","Type":"ContainerStarted","Data":"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4"} Jul 14 18:47:19.876253 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:19.875845 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:19.894144 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:19.894093 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" podStartSLOduration=2.426407871 podStartE2EDuration="2.894076827s" totalImagesPullingTime="467.668956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:47:19.893522907 +0000 UTC m=+2205.279421362" watchObservedRunningTime="2026-07-14 18:47:19.894076827 +0000 UTC m=+2205.279975280" Jul 14 18:47:22.301172 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:22.301140 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:25.886985 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:25.886953 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:47:30.745928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:30.745891 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:30.846955 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:30.846920 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:39.342987 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:39.342952 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:39.443965 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:39.443930 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:41.416348 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.416312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:41.516800 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.516764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:41.679523 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.679437 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:41.679692 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.679670 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" podUID="99aec02a-ce53-4b4b-b452-ededfa8b4603" containerName="maas-api" containerID="cri-o://99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f" gracePeriod=30 Jul 14 18:47:41.717970 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.717942 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:41.934883 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.934805 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:41.988446 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988413 2567 generic.go:356] "Generic (PLEG): container finished" podID="99aec02a-ce53-4b4b-b452-ededfa8b4603" containerID="99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f" exitCode=0 Jul 14 18:47:41.988620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988461 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" event={"ID":"99aec02a-ce53-4b4b-b452-ededfa8b4603","Type":"ContainerDied","Data":"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f"} Jul 14 18:47:41.988620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988467 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" Jul 14 18:47:41.988620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7" event={"ID":"99aec02a-ce53-4b4b-b452-ededfa8b4603","Type":"ContainerDied","Data":"b59a028722576d5928b1efd248e5a4b3c0f81a4f398c19fdedcaabff5f9f0504"} Jul 14 18:47:41.988620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988577 2567 scope.go:117] "RemoveContainer" containerID="99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f" Jul 14 18:47:41.988888 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz4cm\" (UniqueName: \"kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm\") pod \"99aec02a-ce53-4b4b-b452-ededfa8b4603\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " Jul 14 18:47:41.988975 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988926 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") pod \"99aec02a-ce53-4b4b-b452-ededfa8b4603\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " Jul 14 18:47:41.989033 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.988990 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca\") pod \"99aec02a-ce53-4b4b-b452-ededfa8b4603\" (UID: \"99aec02a-ce53-4b4b-b452-ededfa8b4603\") " Jul 14 18:47:41.989391 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.989365 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99aec02a-ce53-4b4b-b452-ededfa8b4603" (UID: "99aec02a-ce53-4b4b-b452-ededfa8b4603"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:47:41.991071 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.991045 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm" (OuterVolumeSpecName: "kube-api-access-jz4cm") pod "99aec02a-ce53-4b4b-b452-ededfa8b4603" (UID: "99aec02a-ce53-4b4b-b452-ededfa8b4603"). InnerVolumeSpecName "kube-api-access-jz4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:47:41.991158 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:41.991104 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99aec02a-ce53-4b4b-b452-ededfa8b4603" (UID: "99aec02a-ce53-4b4b-b452-ededfa8b4603"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:47:42.009112 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.009088 2567 scope.go:117] "RemoveContainer" containerID="99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f" Jul 14 18:47:42.009454 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:47:42.009428 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f\": container with ID starting with 99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f not found: ID does not exist" containerID="99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f" Jul 14 18:47:42.009505 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.009464 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f"} err="failed to get container status \"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f\": rpc error: code = NotFound desc = could not find container \"99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f\": container with ID starting with 99690d3f48662b6183f52c17818cef4f66a4cf7ab22751aaebb96c8a499d1f4f not found: ID does not exist" Jul 14 18:47:42.089894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.089864 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99aec02a-ce53-4b4b-b452-ededfa8b4603-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:47:42.089894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.089889 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4cm\" (UniqueName: \"kubernetes.io/projected/99aec02a-ce53-4b4b-b452-ededfa8b4603-kube-api-access-jz4cm\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:47:42.089894 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.089901 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99aec02a-ce53-4b4b-b452-ededfa8b4603-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:47:42.319083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.319042 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:42.323109 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:42.323079 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fade18-69fdd96979-cmsb7"] Jul 14 18:47:43.931575 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:43.931543 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99aec02a-ce53-4b4b-b452-ededfa8b4603" path="/var/lib/kubelet/pods/99aec02a-ce53-4b4b-b452-ededfa8b4603/volumes" Jul 14 18:47:45.706885 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:45.706850 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:52.066476 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:52.066446 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:52.169395 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:52.169360 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:56.458893 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:56.458807 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:56.718357 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:56.718265 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:57.426433 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:57.426396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:57.531189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:47:57.531152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:07.381953 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.381919 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:48:07.382416 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.382143 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" podUID="c474453c-f466-4120-a059-a595d24d399f" containerName="maas-api" containerID="cri-o://05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4" gracePeriod=30 Jul 14 18:48:07.406453 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.406423 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:07.626006 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.625980 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:48:07.725108 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.725015 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m5sk\" (UniqueName: \"kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk\") pod \"c474453c-f466-4120-a059-a595d24d399f\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " Jul 14 18:48:07.725265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.725119 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") pod \"c474453c-f466-4120-a059-a595d24d399f\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " Jul 14 18:48:07.725265 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.725184 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca\") pod \"c474453c-f466-4120-a059-a595d24d399f\" (UID: \"c474453c-f466-4120-a059-a595d24d399f\") " Jul 14 18:48:07.725565 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.725538 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c474453c-f466-4120-a059-a595d24d399f" (UID: "c474453c-f466-4120-a059-a595d24d399f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:48:07.727169 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.727145 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk" (OuterVolumeSpecName: "kube-api-access-2m5sk") pod "c474453c-f466-4120-a059-a595d24d399f" (UID: "c474453c-f466-4120-a059-a595d24d399f"). InnerVolumeSpecName "kube-api-access-2m5sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:48:07.727263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.727179 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c474453c-f466-4120-a059-a595d24d399f" (UID: "c474453c-f466-4120-a059-a595d24d399f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:48:07.826072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.826030 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c474453c-f466-4120-a059-a595d24d399f-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:48:07.826072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.826064 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c474453c-f466-4120-a059-a595d24d399f-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:48:07.826072 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:07.826078 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m5sk\" (UniqueName: \"kubernetes.io/projected/c474453c-f466-4120-a059-a595d24d399f-kube-api-access-2m5sk\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:48:08.120671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.120621 2567 generic.go:356] "Generic (PLEG): container finished" podID="c474453c-f466-4120-a059-a595d24d399f" containerID="05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4" exitCode=0 Jul 14 18:48:08.120671 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.120666 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" Jul 14 18:48:08.120918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.120705 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" event={"ID":"c474453c-f466-4120-a059-a595d24d399f","Type":"ContainerDied","Data":"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4"} Jul 14 18:48:08.120918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.120770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622" event={"ID":"c474453c-f466-4120-a059-a595d24d399f","Type":"ContainerDied","Data":"d7b3ff440fbf377a5711ea8dacfcfb9e1360d25d84da8f8e0fd0b560602c5032"} Jul 14 18:48:08.120918 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.120798 2567 scope.go:117] "RemoveContainer" containerID="05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4" Jul 14 18:48:08.130132 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.130114 2567 scope.go:117] "RemoveContainer" containerID="05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4" Jul 14 18:48:08.130546 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:48:08.130520 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4\": container with ID starting with 05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4 not found: ID does not exist" containerID="05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4" Jul 14 18:48:08.130620 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.130559 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4"} err="failed to get container status \"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4\": rpc error: code = NotFound desc = could not find container \"05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4\": container with ID starting with 05729d4722d51e81ac7a0b3e7ef1ee41f56f52f3f39421acb51bdf8d3911dff4 not found: ID does not exist" Jul 14 18:48:08.147203 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.147172 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:48:08.151037 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:08.151014 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fde6bc-6948745577-s4622"] Jul 14 18:48:09.931088 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:09.931051 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c474453c-f466-4120-a059-a595d24d399f" path="/var/lib/kubelet/pods/c474453c-f466-4120-a059-a595d24d399f/volumes" Jul 14 18:48:11.951186 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:11.951154 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:18.450953 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:18.450908 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:18.550450 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:18.550410 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:19.406328 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:19.406298 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:19.507093 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:19.507058 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:22.791001 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:22.790967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:23.058510 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:23.058419 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:23.430245 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:23.430164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:23.765448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:23.765391 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:23.837304 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:23.837253 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:23.932717 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:23.932685 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:26.550235 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.550205 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:26.641613 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.641578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:26.643244 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643217 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:48:26.643669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643652 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99aec02a-ce53-4b4b-b452-ededfa8b4603" containerName="maas-api" Jul 14 18:48:26.643751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643671 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99aec02a-ce53-4b4b-b452-ededfa8b4603" containerName="maas-api" Jul 14 18:48:26.643751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643687 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c474453c-f466-4120-a059-a595d24d399f" containerName="maas-api" Jul 14 18:48:26.643751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643695 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c474453c-f466-4120-a059-a595d24d399f" containerName="maas-api" Jul 14 18:48:26.643923 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643856 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c474453c-f466-4120-a059-a595d24d399f" containerName="maas-api" Jul 14 18:48:26.643923 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.643872 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99aec02a-ce53-4b4b-b452-ededfa8b4603" containerName="maas-api" Jul 14 18:48:26.647194 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.647174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.649220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.649201 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5bb958-serving-cert\"" Jul 14 18:48:26.658080 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.658057 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:48:26.704160 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.704112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.704368 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.704184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.704368 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.704289 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsh2k\" (UniqueName: \"kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.707382 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.707346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:26.805151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.805069 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsh2k\" (UniqueName: \"kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.805151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.805135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.805398 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.805184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.805792 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.805766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.807506 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.807486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.814077 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.814055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsh2k\" (UniqueName: \"kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k\") pod \"maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:26.853178 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.853132 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:26.958899 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:26.958865 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:27.099418 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:27.099394 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:48:27.184639 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:27.184604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:27.216008 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:27.215972 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" event={"ID":"87f689ce-bcb0-4a66-9257-c491625c30ea","Type":"ContainerStarted","Data":"9ef4105bb9c8727639e0dcc6addf4617edf5d83b09d55efe81e64f517000951d"} Jul 14 18:48:27.285189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:27.285145 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:28.222251 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:28.222212 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" event={"ID":"87f689ce-bcb0-4a66-9257-c491625c30ea","Type":"ContainerStarted","Data":"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f"} Jul 14 18:48:28.222702 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:28.222405 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:28.241488 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:28.241427 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" podStartSLOduration=1.876714808 podStartE2EDuration="2.241407059s" totalImagesPullingTime="364.692251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:28.240945526 +0000 UTC m=+2273.626843984" watchObservedRunningTime="2026-07-14 18:48:28.241407059 +0000 UTC m=+2273.627305515" Jul 14 18:48:31.108312 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:31.108282 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:31.477852 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:31.477800 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:31.833475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:31.833394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:32.463734 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:32.463702 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:34.232637 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:34.232602 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:48:35.023245 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.023206 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:35.103374 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.103338 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:48:35.104108 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.104082 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:35.107229 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.107210 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.109221 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.109202 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-41a771-serving-cert\"" Jul 14 18:48:35.118234 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.118208 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:48:35.168900 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.168849 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:35.174037 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.174013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.174159 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.174064 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25x9\" (UniqueName: \"kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.174198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.174162 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.275477 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.275390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.275851 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.275478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.275851 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.275536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25x9\" (UniqueName: \"kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.276322 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.276296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.276787 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.276759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:35.278122 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.278101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.284494 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.284471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25x9\" (UniqueName: \"kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9\") pod \"maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.419770 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.419735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:35.560977 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.560953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:48:35.563736 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:48:35.563705 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff37588e_aebb_4d20_91e9_023fbdebeb61.slice/crio-2d8d91a452132ba5ebb7f76022518ec02cf7d096e66244015b460ed19130a675 WatchSource:0}: Error finding container 2d8d91a452132ba5ebb7f76022518ec02cf7d096e66244015b460ed19130a675: Status 404 returned error can't find the container with id 2d8d91a452132ba5ebb7f76022518ec02cf7d096e66244015b460ed19130a675 Jul 14 18:48:35.722734 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.722699 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:35.823298 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:35.823212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:36.264142 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:36.264104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" event={"ID":"ff37588e-aebb-4d20-91e9-023fbdebeb61","Type":"ContainerStarted","Data":"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172"} Jul 14 18:48:36.264294 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:36.264164 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:36.264294 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:36.264183 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" event={"ID":"ff37588e-aebb-4d20-91e9-023fbdebeb61","Type":"ContainerStarted","Data":"2d8d91a452132ba5ebb7f76022518ec02cf7d096e66244015b460ed19130a675"} Jul 14 18:48:36.285669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:36.285619 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" podStartSLOduration=0.884109017 podStartE2EDuration="1.285604468s" totalImagesPullingTime="401.495451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:36.28352025 +0000 UTC m=+2281.669418705" watchObservedRunningTime="2026-07-14 18:48:36.285604468 +0000 UTC m=+2281.671502921" Jul 14 18:48:39.557102 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:39.557059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:42.274179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:42.274151 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:48:49.184239 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:49.184188 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:49.285477 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:49.285446 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:54.863733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:54.863701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:56.589709 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:56.589677 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:56.685393 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:56.685356 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:57.284875 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:48:57.284847 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:26.441259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:49:26.441178 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:36.145420 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:50:36.145313 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:50:36.149760 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:50:36.149738 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:50:55.580906 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:50:55.580870 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:58.069386 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:50:58.069307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:15.693836 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:15.693776 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:22.463665 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:22.463627 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:23.292411 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:23.292367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:29.895763 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:29.895721 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:39.286500 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:39.286452 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:39.387581 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:39.387548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:56.384117 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:51:56.384061 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:51.449105 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:52:51.449070 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:52.270489 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:52:52.270447 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:56.590176 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:52:56.590131 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:08.173617 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:08.173560 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:20.287259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:20.287222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:20.384495 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:20.384458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:32.670836 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:32.670729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:50.025523 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:50.025482 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:50.628655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:50.628607 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:57.539979 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:57.539892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:57.640648 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:53:57.640598 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:15.291373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:15.291339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:16.184485 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:16.184450 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:17.285542 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.285510 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850247344"] Jul 14 18:54:17.289151 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.289133 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:17.291287 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.291263 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sj9r5\"" Jul 14 18:54:17.297650 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.297628 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850247344"] Jul 14 18:54:17.333513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.333479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfzw\" (UniqueName: \"kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw\") pod \"test-curl-7699-140137850247344\" (UID: \"d6938948-2f48-4b5a-a5d8-ca411d256d05\") " pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:17.434451 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.434403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfzw\" (UniqueName: \"kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw\") pod \"test-curl-7699-140137850247344\" (UID: \"d6938948-2f48-4b5a-a5d8-ca411d256d05\") " pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:17.444935 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.444900 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfzw\" (UniqueName: \"kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw\") pod \"test-curl-7699-140137850247344\" (UID: \"d6938948-2f48-4b5a-a5d8-ca411d256d05\") " pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:17.598851 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.598750 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:17.746869 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.746843 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850247344"] Jul 14 18:54:17.749265 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:17.749240 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6938948_2f48_4b5a_a5d8_ca411d256d05.slice/crio-cb1234d7d99c59a4f4e50040ce59e3d6e3042d2ee759ccee4dfd8ece1062f9fc WatchSource:0}: Error finding container cb1234d7d99c59a4f4e50040ce59e3d6e3042d2ee759ccee4dfd8ece1062f9fc: Status 404 returned error can't find the container with id cb1234d7d99c59a4f4e50040ce59e3d6e3042d2ee759ccee4dfd8ece1062f9fc Jul 14 18:54:17.750550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:17.750533 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:54:18.014426 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:18.014384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850247344" event={"ID":"d6938948-2f48-4b5a-a5d8-ca411d256d05","Type":"ContainerStarted","Data":"cb1234d7d99c59a4f4e50040ce59e3d6e3042d2ee759ccee4dfd8ece1062f9fc"} Jul 14 18:54:19.021427 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:19.021386 2567 generic.go:356] "Generic (PLEG): container finished" podID="d6938948-2f48-4b5a-a5d8-ca411d256d05" containerID="86f0f755701a48fe77277275e8da77a9b898c39ccf093a31c8ebcc33fe543e5f" exitCode=0 Jul 14 18:54:19.021797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:19.021470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850247344" event={"ID":"d6938948-2f48-4b5a-a5d8-ca411d256d05","Type":"ContainerDied","Data":"86f0f755701a48fe77277275e8da77a9b898c39ccf093a31c8ebcc33fe543e5f"} Jul 14 18:54:20.161275 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.161251 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:20.184228 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.184204 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850247344_d6938948-2f48-4b5a-a5d8-ca411d256d05/test-curl-7699-140137850247344/0.log" Jul 14 18:54:20.212053 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.212028 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850247344"] Jul 14 18:54:20.219270 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.219246 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850247344"] Jul 14 18:54:20.261178 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.261146 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfzw\" (UniqueName: \"kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw\") pod \"d6938948-2f48-4b5a-a5d8-ca411d256d05\" (UID: \"d6938948-2f48-4b5a-a5d8-ca411d256d05\") " Jul 14 18:54:20.263204 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.263179 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw" (OuterVolumeSpecName: "kube-api-access-tlfzw") pod "d6938948-2f48-4b5a-a5d8-ca411d256d05" (UID: "d6938948-2f48-4b5a-a5d8-ca411d256d05"). InnerVolumeSpecName "kube-api-access-tlfzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:20.359327 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.359242 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850247600"] Jul 14 18:54:20.359676 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.359663 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6938948-2f48-4b5a-a5d8-ca411d256d05" containerName="test-curl-7699-140137850247344" Jul 14 18:54:20.359723 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.359678 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6938948-2f48-4b5a-a5d8-ca411d256d05" containerName="test-curl-7699-140137850247344" Jul 14 18:54:20.359764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.359754 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6938948-2f48-4b5a-a5d8-ca411d256d05" containerName="test-curl-7699-140137850247344" Jul 14 18:54:20.362475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.362454 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfzw\" (UniqueName: \"kubernetes.io/projected/d6938948-2f48-4b5a-a5d8-ca411d256d05-kube-api-access-tlfzw\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:20.362743 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.362728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:20.371292 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.371264 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850247600"] Jul 14 18:54:20.463553 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.463513 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh9s\" (UniqueName: \"kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s\") pod \"test-curl-7699-140137850247600\" (UID: \"bd01c0c9-240f-45a1-b430-e03fc6d46316\") " pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:20.564719 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.564673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh9s\" (UniqueName: \"kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s\") pod \"test-curl-7699-140137850247600\" (UID: \"bd01c0c9-240f-45a1-b430-e03fc6d46316\") " pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:20.574442 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.574413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh9s\" (UniqueName: \"kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s\") pod \"test-curl-7699-140137850247600\" (UID: \"bd01c0c9-240f-45a1-b430-e03fc6d46316\") " pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:20.672892 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.672762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:20.808164 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:20.808140 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850247600"] Jul 14 18:54:20.810710 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:20.810680 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd01c0c9_240f_45a1_b430_e03fc6d46316.slice/crio-e4d614fc47803e388c01688391651c4ab4e3b271f3b1115126da10e65bfe4b5e WatchSource:0}: Error finding container e4d614fc47803e388c01688391651c4ab4e3b271f3b1115126da10e65bfe4b5e: Status 404 returned error can't find the container with id e4d614fc47803e388c01688391651c4ab4e3b271f3b1115126da10e65bfe4b5e Jul 14 18:54:21.032032 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:21.031993 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850247600" event={"ID":"bd01c0c9-240f-45a1-b430-e03fc6d46316","Type":"ContainerStarted","Data":"e4d614fc47803e388c01688391651c4ab4e3b271f3b1115126da10e65bfe4b5e"} Jul 14 18:54:21.033323 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:21.033293 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1234d7d99c59a4f4e50040ce59e3d6e3042d2ee759ccee4dfd8ece1062f9fc" Jul 14 18:54:21.033454 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:21.033327 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:21.043469 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:54:21.043439 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-140137850247344\" is forbidden: User \"system:node:ip-10-0-131-214.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-214.ec2.internal' and this object" podUID="d6938948-2f48-4b5a-a5d8-ca411d256d05" pod="opendatahub/test-curl-7699-140137850247344" Jul 14 18:54:21.930951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:21.930911 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6938948-2f48-4b5a-a5d8-ca411d256d05" path="/var/lib/kubelet/pods/d6938948-2f48-4b5a-a5d8-ca411d256d05/volumes" Jul 14 18:54:22.039917 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:22.039879 2567 generic.go:356] "Generic (PLEG): container finished" podID="bd01c0c9-240f-45a1-b430-e03fc6d46316" containerID="392b03c7ae113928e4c1d6c125a1417f03285ef4dd8dbf8994cb5a5526e332b8" exitCode=0 Jul 14 18:54:22.040083 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:22.039966 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850247600" event={"ID":"bd01c0c9-240f-45a1-b430-e03fc6d46316","Type":"ContainerDied","Data":"392b03c7ae113928e4c1d6c125a1417f03285ef4dd8dbf8994cb5a5526e332b8"} Jul 14 18:54:23.174467 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.174443 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:23.200798 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.200744 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850247600_bd01c0c9-240f-45a1-b430-e03fc6d46316/test-curl-7699-140137850247600/0.log" Jul 14 18:54:23.225693 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.225655 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850247600"] Jul 14 18:54:23.230457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.230424 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850247600"] Jul 14 18:54:23.290534 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.290493 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnh9s\" (UniqueName: \"kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s\") pod \"bd01c0c9-240f-45a1-b430-e03fc6d46316\" (UID: \"bd01c0c9-240f-45a1-b430-e03fc6d46316\") " Jul 14 18:54:23.292551 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.292529 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s" (OuterVolumeSpecName: "kube-api-access-xnh9s") pod "bd01c0c9-240f-45a1-b430-e03fc6d46316" (UID: "bd01c0c9-240f-45a1-b430-e03fc6d46316"). InnerVolumeSpecName "kube-api-access-xnh9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:23.391600 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.391561 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh9s\" (UniqueName: \"kubernetes.io/projected/bd01c0c9-240f-45a1-b430-e03fc6d46316-kube-api-access-xnh9s\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:23.426257 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.426218 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:23.426694 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.426682 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd01c0c9-240f-45a1-b430-e03fc6d46316" containerName="test-curl-7699-140137850247600" Jul 14 18:54:23.426737 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.426696 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd01c0c9-240f-45a1-b430-e03fc6d46316" containerName="test-curl-7699-140137850247600" Jul 14 18:54:23.426776 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.426765 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd01c0c9-240f-45a1-b430-e03fc6d46316" containerName="test-curl-7699-140137850247600" Jul 14 18:54:23.429716 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.429700 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:23.438373 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.438346 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:23.492283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.492249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtb2\" (UniqueName: \"kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2\") pod \"test-curl-7699-140137850282032\" (UID: \"63f45e95-cdb6-4971-82ae-b03ce5e7e176\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:23.593097 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.593011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtb2\" (UniqueName: \"kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2\") pod \"test-curl-7699-140137850282032\" (UID: \"63f45e95-cdb6-4971-82ae-b03ce5e7e176\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:23.605415 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.605378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtb2\" (UniqueName: \"kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2\") pod \"test-curl-7699-140137850282032\" (UID: \"63f45e95-cdb6-4971-82ae-b03ce5e7e176\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:23.739583 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.739533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:23.880955 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.880930 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:23.882925 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:23.882893 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63f45e95_cdb6_4971_82ae_b03ce5e7e176.slice/crio-3d23c8e892423f7807d48df0a3e014d8c0b05abf4f33005003c97d5a59be775b WatchSource:0}: Error finding container 3d23c8e892423f7807d48df0a3e014d8c0b05abf4f33005003c97d5a59be775b: Status 404 returned error can't find the container with id 3d23c8e892423f7807d48df0a3e014d8c0b05abf4f33005003c97d5a59be775b Jul 14 18:54:23.936700 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:23.936669 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd01c0c9-240f-45a1-b430-e03fc6d46316" path="/var/lib/kubelet/pods/bd01c0c9-240f-45a1-b430-e03fc6d46316/volumes" Jul 14 18:54:24.051067 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:24.051029 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850282032" event={"ID":"63f45e95-cdb6-4971-82ae-b03ce5e7e176","Type":"ContainerStarted","Data":"3d23c8e892423f7807d48df0a3e014d8c0b05abf4f33005003c97d5a59be775b"} Jul 14 18:54:24.052189 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:24.052170 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850247600" Jul 14 18:54:24.052281 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:24.052203 2567 scope.go:117] "RemoveContainer" containerID="392b03c7ae113928e4c1d6c125a1417f03285ef4dd8dbf8994cb5a5526e332b8" Jul 14 18:54:25.059198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:25.059148 2567 generic.go:356] "Generic (PLEG): container finished" podID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" containerID="86ac028398b387b932e15bd3b94a51da6654c97c6971f78e548a883b607e495b" exitCode=0 Jul 14 18:54:25.059730 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:25.059263 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850282032" event={"ID":"63f45e95-cdb6-4971-82ae-b03ce5e7e176","Type":"ContainerDied","Data":"86ac028398b387b932e15bd3b94a51da6654c97c6971f78e548a883b607e495b"} Jul 14 18:54:26.196270 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.196242 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.221839 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.221802 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850282032_63f45e95-cdb6-4971-82ae-b03ce5e7e176/test-curl-7699-140137850282032/0.log" Jul 14 18:54:26.251349 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.251311 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:26.258228 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.258196 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:26.319747 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.319649 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shtb2\" (UniqueName: \"kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2\") pod \"63f45e95-cdb6-4971-82ae-b03ce5e7e176\" (UID: \"63f45e95-cdb6-4971-82ae-b03ce5e7e176\") " Jul 14 18:54:26.321788 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.321761 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2" (OuterVolumeSpecName: "kube-api-access-shtb2") pod "63f45e95-cdb6-4971-82ae-b03ce5e7e176" (UID: "63f45e95-cdb6-4971-82ae-b03ce5e7e176"). InnerVolumeSpecName "kube-api-access-shtb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:26.403722 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.403686 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:26.404139 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.404125 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" containerName="test-curl-7699-140137850282032" Jul 14 18:54:26.404184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.404140 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" containerName="test-curl-7699-140137850282032" Jul 14 18:54:26.404220 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.404199 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" containerName="test-curl-7699-140137850282032" Jul 14 18:54:26.407209 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.407194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.412306 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.412278 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:26.420829 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.420790 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtb2\" (UniqueName: \"kubernetes.io/projected/63f45e95-cdb6-4971-82ae-b03ce5e7e176-kube-api-access-shtb2\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:26.521513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.521476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q4c\" (UniqueName: \"kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c\") pod \"test-curl-7699-140137850282032\" (UID: \"94e20799-3f4c-437b-b2f8-da9cacc0cfb1\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.622154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.622063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q4c\" (UniqueName: \"kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c\") pod \"test-curl-7699-140137850282032\" (UID: \"94e20799-3f4c-437b-b2f8-da9cacc0cfb1\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.631506 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.631476 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q4c\" (UniqueName: \"kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c\") pod \"test-curl-7699-140137850282032\" (UID: \"94e20799-3f4c-437b-b2f8-da9cacc0cfb1\") " pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.718179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.718148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:26.857797 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:26.857770 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:26.859966 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:26.859937 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e20799_3f4c_437b_b2f8_da9cacc0cfb1.slice/crio-9335f1970f71d91b7edc185c04b8e26bb826c52ddaf20c45cd243cf92ed152ac WatchSource:0}: Error finding container 9335f1970f71d91b7edc185c04b8e26bb826c52ddaf20c45cd243cf92ed152ac: Status 404 returned error can't find the container with id 9335f1970f71d91b7edc185c04b8e26bb826c52ddaf20c45cd243cf92ed152ac Jul 14 18:54:27.071952 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:27.071905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850282032" event={"ID":"94e20799-3f4c-437b-b2f8-da9cacc0cfb1","Type":"ContainerStarted","Data":"9335f1970f71d91b7edc185c04b8e26bb826c52ddaf20c45cd243cf92ed152ac"} Jul 14 18:54:27.073094 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:27.073076 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:27.073175 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:27.073111 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d23c8e892423f7807d48df0a3e014d8c0b05abf4f33005003c97d5a59be775b" Jul 14 18:54:27.089572 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:27.089534 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7699-140137850282032" oldPodUID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" podUID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" Jul 14 18:54:27.931259 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:27.931226 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f45e95-cdb6-4971-82ae-b03ce5e7e176" path="/var/lib/kubelet/pods/63f45e95-cdb6-4971-82ae-b03ce5e7e176/volumes" Jul 14 18:54:28.080226 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:28.080187 2567 generic.go:356] "Generic (PLEG): container finished" podID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" containerID="d86df0275a0a78d544e4eba1cfb86cfb45b63034e45ad36c643e4af0a58ee293" exitCode=0 Jul 14 18:54:28.080399 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:28.080244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850282032" event={"ID":"94e20799-3f4c-437b-b2f8-da9cacc0cfb1","Type":"ContainerDied","Data":"d86df0275a0a78d544e4eba1cfb86cfb45b63034e45ad36c643e4af0a58ee293"} Jul 14 18:54:29.222011 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.221985 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:29.244897 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.244869 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850282032_94e20799-3f4c-437b-b2f8-da9cacc0cfb1/test-curl-7699-140137850282032/0.log" Jul 14 18:54:29.278517 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.278488 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:29.283628 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.283593 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850282032"] Jul 14 18:54:29.348655 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.348560 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7q4c\" (UniqueName: \"kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c\") pod \"94e20799-3f4c-437b-b2f8-da9cacc0cfb1\" (UID: \"94e20799-3f4c-437b-b2f8-da9cacc0cfb1\") " Jul 14 18:54:29.350625 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.350604 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c" (OuterVolumeSpecName: "kube-api-access-j7q4c") pod "94e20799-3f4c-437b-b2f8-da9cacc0cfb1" (UID: "94e20799-3f4c-437b-b2f8-da9cacc0cfb1"). InnerVolumeSpecName "kube-api-access-j7q4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:29.428250 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.428211 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:29.428690 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.428676 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" containerName="test-curl-7699-140137850282032" Jul 14 18:54:29.428760 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.428693 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" containerName="test-curl-7699-140137850282032" Jul 14 18:54:29.428796 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.428767 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" containerName="test-curl-7699-140137850282032" Jul 14 18:54:29.431796 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.431780 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:29.439438 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.439416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:29.449592 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.449565 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7q4c\" (UniqueName: \"kubernetes.io/projected/94e20799-3f4c-437b-b2f8-da9cacc0cfb1-kube-api-access-j7q4c\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:29.550171 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.550127 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bb9\" (UniqueName: \"kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9\") pod \"test-curl-7699-140137850283040\" (UID: \"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:29.651157 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.651078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bb9\" (UniqueName: \"kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9\") pod \"test-curl-7699-140137850283040\" (UID: \"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:29.659889 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.659859 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bb9\" (UniqueName: \"kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9\") pod \"test-curl-7699-140137850283040\" (UID: \"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:29.742687 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.742650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:29.876672 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.876646 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:29.879018 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:29.878992 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04bdec2_2cdb_4081_acdb_bc1e5207fbdc.slice/crio-7452cf994058a92baf953018ac2eea63d5b9bd01a7f19fb2c6f3f0469199068e WatchSource:0}: Error finding container 7452cf994058a92baf953018ac2eea63d5b9bd01a7f19fb2c6f3f0469199068e: Status 404 returned error can't find the container with id 7452cf994058a92baf953018ac2eea63d5b9bd01a7f19fb2c6f3f0469199068e Jul 14 18:54:29.933863 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:29.933836 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e20799-3f4c-437b-b2f8-da9cacc0cfb1" path="/var/lib/kubelet/pods/94e20799-3f4c-437b-b2f8-da9cacc0cfb1/volumes" Jul 14 18:54:30.091080 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:30.091053 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850282032" Jul 14 18:54:30.091296 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:30.091100 2567 scope.go:117] "RemoveContainer" containerID="d86df0275a0a78d544e4eba1cfb86cfb45b63034e45ad36c643e4af0a58ee293" Jul 14 18:54:30.092481 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:30.092275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850283040" event={"ID":"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc","Type":"ContainerStarted","Data":"7452cf994058a92baf953018ac2eea63d5b9bd01a7f19fb2c6f3f0469199068e"} Jul 14 18:54:31.099891 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:31.099856 2567 generic.go:356] "Generic (PLEG): container finished" podID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" containerID="f51605e10893ddb750cbb35576aeaa76001340e5b598d495fb63c43f5deddbf3" exitCode=0 Jul 14 18:54:31.100289 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:31.099956 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850283040" event={"ID":"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc","Type":"ContainerDied","Data":"f51605e10893ddb750cbb35576aeaa76001340e5b598d495fb63c43f5deddbf3"} Jul 14 18:54:32.245006 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.244984 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.273162 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.273133 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850283040_f04bdec2-2cdb-4081-acdb-bc1e5207fbdc/test-curl-7699-140137850283040/0.log" Jul 14 18:54:32.299318 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.299286 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:32.304309 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.304280 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:32.379248 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.379154 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5bb9\" (UniqueName: \"kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9\") pod \"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc\" (UID: \"f04bdec2-2cdb-4081-acdb-bc1e5207fbdc\") " Jul 14 18:54:32.381274 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.381243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9" (OuterVolumeSpecName: "kube-api-access-d5bb9") pod "f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" (UID: "f04bdec2-2cdb-4081-acdb-bc1e5207fbdc"). InnerVolumeSpecName "kube-api-access-d5bb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:32.440733 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.440680 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:32.441216 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.441197 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" containerName="test-curl-7699-140137850283040" Jul 14 18:54:32.441327 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.441217 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" containerName="test-curl-7699-140137850283040" Jul 14 18:54:32.441383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.441354 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" containerName="test-curl-7699-140137850283040" Jul 14 18:54:32.444392 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.444371 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.452243 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.452222 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:32.480708 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.480667 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5bb9\" (UniqueName: \"kubernetes.io/projected/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc-kube-api-access-d5bb9\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:32.581381 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.581328 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjth\" (UniqueName: \"kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth\") pod \"test-curl-7699-140137850283040\" (UID: \"d36eabbe-63dc-44fe-a225-36aab5f20453\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.682854 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.682730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjth\" (UniqueName: \"kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth\") pod \"test-curl-7699-140137850283040\" (UID: \"d36eabbe-63dc-44fe-a225-36aab5f20453\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.693480 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.693439 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjth\" (UniqueName: \"kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth\") pod \"test-curl-7699-140137850283040\" (UID: \"d36eabbe-63dc-44fe-a225-36aab5f20453\") " pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.754384 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.754346 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:32.892774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:32.892623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:32.895679 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:54:32.895649 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd36eabbe_63dc_44fe_a225_36aab5f20453.slice/crio-ee7ede4983e37c8990e3e81cf74b3f8accbbadb87a533ca5c2a19a4f042260cf WatchSource:0}: Error finding container ee7ede4983e37c8990e3e81cf74b3f8accbbadb87a533ca5c2a19a4f042260cf: Status 404 returned error can't find the container with id ee7ede4983e37c8990e3e81cf74b3f8accbbadb87a533ca5c2a19a4f042260cf Jul 14 18:54:33.120108 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:33.120067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850283040" event={"ID":"d36eabbe-63dc-44fe-a225-36aab5f20453","Type":"ContainerStarted","Data":"ee7ede4983e37c8990e3e81cf74b3f8accbbadb87a533ca5c2a19a4f042260cf"} Jul 14 18:54:33.121345 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:33.121310 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:33.121475 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:33.121312 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7452cf994058a92baf953018ac2eea63d5b9bd01a7f19fb2c6f3f0469199068e" Jul 14 18:54:33.133412 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:33.133381 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7699-140137850283040" oldPodUID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" podUID="d36eabbe-63dc-44fe-a225-36aab5f20453" Jul 14 18:54:33.931440 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:33.931408 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04bdec2-2cdb-4081-acdb-bc1e5207fbdc" path="/var/lib/kubelet/pods/f04bdec2-2cdb-4081-acdb-bc1e5207fbdc/volumes" Jul 14 18:54:34.127990 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:34.127956 2567 generic.go:356] "Generic (PLEG): container finished" podID="d36eabbe-63dc-44fe-a225-36aab5f20453" containerID="906365173ddd7f3ab5ddadd3ac556efa02c01ec7cf6538d61a8371f72f43b641" exitCode=0 Jul 14 18:54:34.128134 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:34.128039 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-140137850283040" event={"ID":"d36eabbe-63dc-44fe-a225-36aab5f20453","Type":"ContainerDied","Data":"906365173ddd7f3ab5ddadd3ac556efa02c01ec7cf6538d61a8371f72f43b641"} Jul 14 18:54:35.270585 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.270559 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:35.293526 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.293500 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7699-140137850283040_d36eabbe-63dc-44fe-a225-36aab5f20453/test-curl-7699-140137850283040/0.log" Jul 14 18:54:35.319327 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.319281 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:35.326825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.326790 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-140137850283040"] Jul 14 18:54:35.412271 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.412180 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmjth\" (UniqueName: \"kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth\") pod \"d36eabbe-63dc-44fe-a225-36aab5f20453\" (UID: \"d36eabbe-63dc-44fe-a225-36aab5f20453\") " Jul 14 18:54:35.414279 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.414257 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth" (OuterVolumeSpecName: "kube-api-access-cmjth") pod "d36eabbe-63dc-44fe-a225-36aab5f20453" (UID: "d36eabbe-63dc-44fe-a225-36aab5f20453"). InnerVolumeSpecName "kube-api-access-cmjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:35.513385 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.513345 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjth\" (UniqueName: \"kubernetes.io/projected/d36eabbe-63dc-44fe-a225-36aab5f20453-kube-api-access-cmjth\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:35.931605 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:35.931571 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36eabbe-63dc-44fe-a225-36aab5f20453" path="/var/lib/kubelet/pods/d36eabbe-63dc-44fe-a225-36aab5f20453/volumes" Jul 14 18:54:36.139825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:36.139785 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-140137850283040" Jul 14 18:54:36.139825 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:36.139804 2567 scope.go:117] "RemoveContainer" containerID="906365173ddd7f3ab5ddadd3ac556efa02c01ec7cf6538d61a8371f72f43b641" Jul 14 18:54:36.155762 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:36.155733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:36.256118 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:36.256089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:46.083751 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.083712 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:54:46.084221 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.084039 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" podUID="87f689ce-bcb0-4a66-9257-c491625c30ea" containerName="maas-api" containerID="cri-o://edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f" gracePeriod=30 Jul 14 18:54:46.107363 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.107337 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:46.334007 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.333938 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:54:46.423448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.423413 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsh2k\" (UniqueName: \"kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k\") pod \"87f689ce-bcb0-4a66-9257-c491625c30ea\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " Jul 14 18:54:46.423649 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.423490 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls\") pod \"87f689ce-bcb0-4a66-9257-c491625c30ea\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " Jul 14 18:54:46.423649 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.423536 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca\") pod \"87f689ce-bcb0-4a66-9257-c491625c30ea\" (UID: \"87f689ce-bcb0-4a66-9257-c491625c30ea\") " Jul 14 18:54:46.424000 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.423965 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87f689ce-bcb0-4a66-9257-c491625c30ea" (UID: "87f689ce-bcb0-4a66-9257-c491625c30ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:54:46.425704 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.425684 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87f689ce-bcb0-4a66-9257-c491625c30ea" (UID: "87f689ce-bcb0-4a66-9257-c491625c30ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:54:46.425790 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.425709 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k" (OuterVolumeSpecName: "kube-api-access-dsh2k") pod "87f689ce-bcb0-4a66-9257-c491625c30ea" (UID: "87f689ce-bcb0-4a66-9257-c491625c30ea"). InnerVolumeSpecName "kube-api-access-dsh2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:54:46.525513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.525453 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f689ce-bcb0-4a66-9257-c491625c30ea-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:46.525513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.525492 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f689ce-bcb0-4a66-9257-c491625c30ea-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:46.525513 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:46.525509 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsh2k\" (UniqueName: \"kubernetes.io/projected/87f689ce-bcb0-4a66-9257-c491625c30ea-kube-api-access-dsh2k\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:54:47.198199 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.198159 2567 generic.go:356] "Generic (PLEG): container finished" podID="87f689ce-bcb0-4a66-9257-c491625c30ea" containerID="edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f" exitCode=0 Jul 14 18:54:47.198550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.198224 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" Jul 14 18:54:47.198550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.198246 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" event={"ID":"87f689ce-bcb0-4a66-9257-c491625c30ea","Type":"ContainerDied","Data":"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f"} Jul 14 18:54:47.198550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.198308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8" event={"ID":"87f689ce-bcb0-4a66-9257-c491625c30ea","Type":"ContainerDied","Data":"9ef4105bb9c8727639e0dcc6addf4617edf5d83b09d55efe81e64f517000951d"} Jul 14 18:54:47.198550 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.198333 2567 scope.go:117] "RemoveContainer" containerID="edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f" Jul 14 18:54:47.208009 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.207992 2567 scope.go:117] "RemoveContainer" containerID="edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f" Jul 14 18:54:47.208248 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:54:47.208231 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f\": container with ID starting with edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f not found: ID does not exist" containerID="edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f" Jul 14 18:54:47.208297 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.208256 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f"} err="failed to get container status \"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f\": rpc error: code = NotFound desc = could not find container \"edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f\": container with ID starting with edd9013c995431cd0b3cbdb71670cbfbef5a337a19175da7e3677184ef6c2d9f not found: ID does not exist" Jul 14 18:54:47.223435 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.223406 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:54:47.228413 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.228387 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5bb958-77d8b5cc74-klnw8"] Jul 14 18:54:47.931822 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:47.931768 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f689ce-bcb0-4a66-9257-c491625c30ea" path="/var/lib/kubelet/pods/87f689ce-bcb0-4a66-9257-c491625c30ea/volumes" Jul 14 18:54:50.746758 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:50.746722 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:57.892074 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:57.892036 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:58.089173 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:54:58.089137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:01.566434 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:01.566397 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:01.837398 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:01.837301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:02.540209 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:02.540176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:02.641056 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:02.641020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.498168 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.498122 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:55:12.498627 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.498357 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" podUID="ff37588e-aebb-4d20-91e9-023fbdebeb61" containerName="maas-api" containerID="cri-o://09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172" gracePeriod=30 Jul 14 18:55:12.520954 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.520924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.750184 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.750114 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:55:12.876756 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.876716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca\") pod \"ff37588e-aebb-4d20-91e9-023fbdebeb61\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " Jul 14 18:55:12.876940 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.876794 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls\") pod \"ff37588e-aebb-4d20-91e9-023fbdebeb61\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " Jul 14 18:55:12.876940 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.876925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z25x9\" (UniqueName: \"kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9\") pod \"ff37588e-aebb-4d20-91e9-023fbdebeb61\" (UID: \"ff37588e-aebb-4d20-91e9-023fbdebeb61\") " Jul 14 18:55:12.877175 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.877148 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff37588e-aebb-4d20-91e9-023fbdebeb61" (UID: "ff37588e-aebb-4d20-91e9-023fbdebeb61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:12.878928 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.878899 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff37588e-aebb-4d20-91e9-023fbdebeb61" (UID: "ff37588e-aebb-4d20-91e9-023fbdebeb61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:55:12.879052 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.879029 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9" (OuterVolumeSpecName: "kube-api-access-z25x9") pod "ff37588e-aebb-4d20-91e9-023fbdebeb61" (UID: "ff37588e-aebb-4d20-91e9-023fbdebeb61"). InnerVolumeSpecName "kube-api-access-z25x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:12.978344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.978304 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z25x9\" (UniqueName: \"kubernetes.io/projected/ff37588e-aebb-4d20-91e9-023fbdebeb61-kube-api-access-z25x9\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:55:12.978344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.978336 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff37588e-aebb-4d20-91e9-023fbdebeb61-openshift-service-ca\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:55:12.978344 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:12.978348 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff37588e-aebb-4d20-91e9-023fbdebeb61-maas-api-tls\") on node \"ip-10-0-131-214.ec2.internal\" DevicePath \"\"" Jul 14 18:55:13.340481 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.340448 2567 generic.go:356] "Generic (PLEG): container finished" podID="ff37588e-aebb-4d20-91e9-023fbdebeb61" containerID="09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172" exitCode=0 Jul 14 18:55:13.340652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.340494 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" Jul 14 18:55:13.340652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.340517 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" event={"ID":"ff37588e-aebb-4d20-91e9-023fbdebeb61","Type":"ContainerDied","Data":"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172"} Jul 14 18:55:13.340652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.340559 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn" event={"ID":"ff37588e-aebb-4d20-91e9-023fbdebeb61","Type":"ContainerDied","Data":"2d8d91a452132ba5ebb7f76022518ec02cf7d096e66244015b460ed19130a675"} Jul 14 18:55:13.340652 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.340580 2567 scope.go:117] "RemoveContainer" containerID="09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172" Jul 14 18:55:13.351931 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.351907 2567 scope.go:117] "RemoveContainer" containerID="09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172" Jul 14 18:55:13.352208 ip-10-0-131-214 kubenswrapper[2567]: E0714 18:55:13.352185 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172\": container with ID starting with 09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172 not found: ID does not exist" containerID="09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172" Jul 14 18:55:13.352267 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.352217 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172"} err="failed to get container status \"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172\": rpc error: code = NotFound desc = could not find container \"09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172\": container with ID starting with 09d7b44e96ef34bfed48808fd7b865d5ebd13f22d252c6a083583da6190ad172 not found: ID does not exist" Jul 14 18:55:13.367193 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.367165 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:55:13.372321 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.372299 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-41a771-6548bf49c7-cnxxn"] Jul 14 18:55:13.931412 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:13.931376 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff37588e-aebb-4d20-91e9-023fbdebeb61" path="/var/lib/kubelet/pods/ff37588e-aebb-4d20-91e9-023fbdebeb61/volumes" Jul 14 18:55:17.110282 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:17.110248 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:24.017725 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:24.017694 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:24.080842 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:24.080778 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:27.879172 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:27.879088 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:28.170943 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:28.170856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:34.330983 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:34.330948 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcd7b898f-nqprv_63c8ac52-0bb3-43a1-8521-cea6f8d6d661/manager/0.log" Jul 14 18:55:34.873034 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:34.873002 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-86prt_4ef798df-dc29-4dc3-b23a-fc2274e34761/copy-manifests/0.log" Jul 14 18:55:34.880754 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:34.880707 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-86prt_4ef798df-dc29-4dc3-b23a-fc2274e34761/manager/0.log" Jul 14 18:55:35.417951 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:35.417916 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b45ff5dc-gmz2g_0e34e5cf-0cce-45a2-9969-1ba3ad56f1c0/maas-api/0.log" Jul 14 18:55:35.542104 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:35.542077 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734230-97ghb_0c11edfe-f183-491a-9451-40418e15993b/cleanup/0.log" Jul 14 18:55:36.177774 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:36.177667 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:55:36.184628 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:36.184607 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dmgf9_df8b3c37-aeb5-4c19-986f-725b81421e2d/ovn-acl-logging/0.log" Jul 14 18:55:37.319764 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:37.319737 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-v8l6k_ed3bc6b5-5d29-4924-80f8-c0b7a5844d6c/postgres/0.log" Jul 14 18:55:40.288786 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:40.288755 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-fwm48_2f4c8035-e58d-45d4-b82e-513983279c85/kube-auth-proxy/0.log" Jul 14 18:55:40.671827 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:40.671725 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-4dk9s_79ccee40-6189-4489-8087-a3dc68495c21/payload-pre-processing/0.log" Jul 14 18:55:40.807409 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:40.807380 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-gfmmb_29b98be0-12b3-4486-a0ce-ca7627424ecb/payload-processing/0.log" Jul 14 18:55:41.304956 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:41.304925 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z8m6k_a58e0ed6-b9e1-41a0-9680-2f5a39b58c57/main/0.log" Jul 14 18:55:41.676139 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:41.676054 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db22dv9_f00d4bf2-0c86-4825-bc4a-8d1b00213d38/main/0.log" Jul 14 18:55:47.025663 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:47.025629 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:47.097198 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:47.097159 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:50.078154 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:50.078124 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-flxqs_e0e4e0ae-937a-401d-a47c-6691c6926b23/global-pull-secret-syncer/0.log" Jul 14 18:55:50.140674 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:50.140646 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bb8mv_0a213abe-36cc-4531-bcb1-012b5312a6a0/konnectivity-agent/0.log" Jul 14 18:55:50.280212 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:50.280183 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-214.ec2.internal_3a8b05c1a94b1956b000d33abc88ab88/haproxy/0.log" Jul 14 18:55:57.649470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.649447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/alertmanager/0.log" Jul 14 18:55:57.679101 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.679070 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/config-reloader/0.log" Jul 14 18:55:57.710641 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.710618 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/kube-rbac-proxy-web/0.log" Jul 14 18:55:57.738024 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.737997 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/kube-rbac-proxy/0.log" Jul 14 18:55:57.775820 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.775794 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/kube-rbac-proxy-metric/0.log" Jul 14 18:55:57.800794 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.800760 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/prom-label-proxy/0.log" Jul 14 18:55:57.826191 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.826169 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c77b60a3-5ced-41a9-95e9-c24a12203274/init-config-reloader/0.log" Jul 14 18:55:57.902383 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.902307 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xlqh7_3a6b7512-3cbe-4300-ae4f-211c1809c6be/kube-state-metrics/0.log" Jul 14 18:55:57.930180 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.930154 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xlqh7_3a6b7512-3cbe-4300-ae4f-211c1809c6be/kube-rbac-proxy-main/0.log" Jul 14 18:55:57.959629 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.959587 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xlqh7_3a6b7512-3cbe-4300-ae4f-211c1809c6be/kube-rbac-proxy-self/0.log" Jul 14 18:55:57.991677 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:57.991650 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bcb994478-jgh2r_d307dd2d-c841-4155-b8fb-2b67110dfdaa/metrics-server/0.log" Jul 14 18:55:58.019829 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.019791 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-75s4d_eddae40e-6050-4fd9-a04e-bee4ef47ac55/monitoring-plugin/0.log" Jul 14 18:55:58.246398 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.246366 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2d8l_dd053b0f-e96c-4e0a-8669-1f2b2c3ff499/node-exporter/0.log" Jul 14 18:55:58.271011 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.270983 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2d8l_dd053b0f-e96c-4e0a-8669-1f2b2c3ff499/kube-rbac-proxy/0.log" Jul 14 18:55:58.295044 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.295018 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2d8l_dd053b0f-e96c-4e0a-8669-1f2b2c3ff499/init-textfile/0.log" Jul 14 18:55:58.333835 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.333782 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qkwf8_b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe/kube-rbac-proxy-main/0.log" Jul 14 18:55:58.353079 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.353049 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qkwf8_b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe/kube-rbac-proxy-self/0.log" Jul 14 18:55:58.381113 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.381085 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qkwf8_b7832a17-4a68-4cf7-a8ff-8dd90fd9fcbe/openshift-state-metrics/0.log" Jul 14 18:55:58.423855 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.423826 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/prometheus/0.log" Jul 14 18:55:58.450050 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.450020 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/config-reloader/0.log" Jul 14 18:55:58.476115 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.476084 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/thanos-sidecar/0.log" Jul 14 18:55:58.501174 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.501098 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/kube-rbac-proxy-web/0.log" Jul 14 18:55:58.527669 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.527643 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/kube-rbac-proxy/0.log" Jul 14 18:55:58.553179 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.553146 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/kube-rbac-proxy-thanos/0.log" Jul 14 18:55:58.578569 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.578547 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fec06e9f-7f74-4454-834a-3dfd0cc504e7/init-config-reloader/0.log" Jul 14 18:55:58.609940 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.609915 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6ddtx_95ac1ce4-aaa4-4834-817a-8f0f877450bd/prometheus-operator/0.log" Jul 14 18:55:58.632470 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.632447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6ddtx_95ac1ce4-aaa4-4834-817a-8f0f877450bd/kube-rbac-proxy/0.log" Jul 14 18:55:58.711068 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.711041 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5b69564-6whnd_fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6/telemeter-client/0.log" Jul 14 18:55:58.739717 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.739687 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5b69564-6whnd_fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6/reload/0.log" Jul 14 18:55:58.766454 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.766380 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5b69564-6whnd_fa31bb53-e5c3-466e-bf40-e18cbbe7dcf6/kube-rbac-proxy/0.log" Jul 14 18:55:58.798744 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.798716 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/thanos-query/0.log" Jul 14 18:55:58.825066 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.825039 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/kube-rbac-proxy-web/0.log" Jul 14 18:55:58.859026 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.859001 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/kube-rbac-proxy/0.log" Jul 14 18:55:58.881663 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.881635 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/prom-label-proxy/0.log" Jul 14 18:55:58.905432 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.905390 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/kube-rbac-proxy-rules/0.log" Jul 14 18:55:58.930311 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:58.930281 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84457cb55-bgwd5_b2c72ef5-78e9-4877-a5c5-758bfbcdfdb1/kube-rbac-proxy-metrics/0.log" Jul 14 18:55:59.714868 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.714831 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc"] Jul 14 18:55:59.715305 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715289 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f689ce-bcb0-4a66-9257-c491625c30ea" containerName="maas-api" Jul 14 18:55:59.715352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715307 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f689ce-bcb0-4a66-9257-c491625c30ea" containerName="maas-api" Jul 14 18:55:59.715352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715321 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff37588e-aebb-4d20-91e9-023fbdebeb61" containerName="maas-api" Jul 14 18:55:59.715352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715328 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff37588e-aebb-4d20-91e9-023fbdebeb61" containerName="maas-api" Jul 14 18:55:59.715352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715339 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36eabbe-63dc-44fe-a225-36aab5f20453" containerName="test-curl-7699-140137850283040" Jul 14 18:55:59.715352 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715346 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36eabbe-63dc-44fe-a225-36aab5f20453" containerName="test-curl-7699-140137850283040" Jul 14 18:55:59.715503 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715444 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f689ce-bcb0-4a66-9257-c491625c30ea" containerName="maas-api" Jul 14 18:55:59.715503 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715455 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36eabbe-63dc-44fe-a225-36aab5f20453" containerName="test-curl-7699-140137850283040" Jul 14 18:55:59.715503 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.715468 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff37588e-aebb-4d20-91e9-023fbdebeb61" containerName="maas-api" Jul 14 18:55:59.718786 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.718765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.721000 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.720977 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"kube-root-ca.crt\"" Jul 14 18:55:59.721127 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.721016 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-492wj\"/\"openshift-service-ca.crt\"" Jul 14 18:55:59.721497 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.721479 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-492wj\"/\"default-dockercfg-n6svq\"" Jul 14 18:55:59.726973 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.726953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc"] Jul 14 18:55:59.737235 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.737207 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-proc\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.737366 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.737246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-lib-modules\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.737366 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.737322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpn9\" (UniqueName: \"kubernetes.io/projected/ec071833-a39e-458c-ad79-cd4f747632b1-kube-api-access-kdpn9\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.737457 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.737440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-sys\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.737493 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.737481 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-podres\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838389 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-sys\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-podres\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-proc\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-sys\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-lib-modules\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838555 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpn9\" (UniqueName: \"kubernetes.io/projected/ec071833-a39e-458c-ad79-cd4f747632b1-kube-api-access-kdpn9\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838721 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-podres\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838721 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-proc\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.838721 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.838713 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec071833-a39e-458c-ad79-cd4f747632b1-lib-modules\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:55:59.847291 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:55:59.847262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpn9\" (UniqueName: \"kubernetes.io/projected/ec071833-a39e-458c-ad79-cd4f747632b1-kube-api-access-kdpn9\") pod \"perf-node-gather-daemonset-k7qhc\" (UID: \"ec071833-a39e-458c-ad79-cd4f747632b1\") " pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:56:00.030236 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.030203 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:56:00.163169 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.163070 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc"] Jul 14 18:56:00.165912 ip-10-0-131-214 kubenswrapper[2567]: W0714 18:56:00.165882 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podec071833_a39e_458c_ad79_cd4f747632b1.slice/crio-df9b5ba0fd2744a9433cc833249e084f567df090e463912c747378f6a003f6d5 WatchSource:0}: Error finding container df9b5ba0fd2744a9433cc833249e084f567df090e463912c747378f6a003f6d5: Status 404 returned error can't find the container with id df9b5ba0fd2744a9433cc833249e084f567df090e463912c747378f6a003f6d5 Jul 14 18:56:00.588123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.588028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" event={"ID":"ec071833-a39e-458c-ad79-cd4f747632b1","Type":"ContainerStarted","Data":"864f7a498814477a094bf5d5658dd94a06bfa0d900570ae65664165c149515bb"} Jul 14 18:56:00.588123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.588079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" event={"ID":"ec071833-a39e-458c-ad79-cd4f747632b1","Type":"ContainerStarted","Data":"df9b5ba0fd2744a9433cc833249e084f567df090e463912c747378f6a003f6d5"} Jul 14 18:56:00.588123 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.588112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:56:00.605343 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:00.605287 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" podStartSLOduration=1.605270246 podStartE2EDuration="1.605270246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:00.604182341 +0000 UTC m=+2725.990080806" watchObservedRunningTime="2026-07-14 18:56:00.605270246 +0000 UTC m=+2725.991168699" Jul 14 18:56:01.316647 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:01.316617 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fd775db8c-7bszs_02d7861e-996b-435a-ba4a-7a003620b178/console/0.log" Jul 14 18:56:03.160263 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:03.160234 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jllgd_75db477c-1c43-443c-b5cc-3ce65f8e89a0/dns/0.log" Jul 14 18:56:03.190448 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:03.190417 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jllgd_75db477c-1c43-443c-b5cc-3ce65f8e89a0/kube-rbac-proxy/0.log" Jul 14 18:56:03.333305 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:03.333277 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxk2x_f330b323-6e28-48a7-b5b9-78d59ac1bf97/dns-node-resolver/0.log" Jul 14 18:56:03.991407 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:03.991372 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vw8tj_55f5a69d-b410-48eb-af6a-77592914a97d/node-ca/0.log" Jul 14 18:56:05.204425 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:05.204395 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-fwm48_2f4c8035-e58d-45d4-b82e-513983279c85/kube-auth-proxy/0.log" Jul 14 18:56:05.328795 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:05.328763 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-4dk9s_79ccee40-6189-4489-8087-a3dc68495c21/payload-pre-processing/0.log" Jul 14 18:56:05.361924 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:05.361893 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-gfmmb_29b98be0-12b3-4486-a0ce-ca7627424ecb/payload-processing/0.log" Jul 14 18:56:06.127586 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:06.127563 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8st4g_27fe26db-4700-4c31-9f83-219402644385/serve-healthcheck-canary/0.log" Jul 14 18:56:06.603283 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:06.603254 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-492wj/perf-node-gather-daemonset-k7qhc" Jul 14 18:56:06.900913 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:06.900807 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4jp2_22bab0bb-fb05-4ea8-9ee0-8e57a0958075/kube-rbac-proxy/0.log" Jul 14 18:56:06.927048 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:06.927018 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4jp2_22bab0bb-fb05-4ea8-9ee0-8e57a0958075/exporter/0.log" Jul 14 18:56:06.954590 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:06.954564 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4jp2_22bab0bb-fb05-4ea8-9ee0-8e57a0958075/extractor/0.log" Jul 14 18:56:09.646778 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:09.646703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcd7b898f-nqprv_63c8ac52-0bb3-43a1-8521-cea6f8d6d661/manager/0.log" Jul 14 18:56:09.878623 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:09.878592 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-86prt_4ef798df-dc29-4dc3-b23a-fc2274e34761/manager/0.log" Jul 14 18:56:09.903167 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:09.903087 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-557b7f646c-86prt_4ef798df-dc29-4dc3-b23a-fc2274e34761/copy-manifests/0.log" Jul 14 18:56:10.437602 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:10.437569 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-fwm48_2f4c8035-e58d-45d4-b82e-513983279c85/kube-auth-proxy/0.log" Jul 14 18:56:10.527187 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:10.527137 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-4dk9s_79ccee40-6189-4489-8087-a3dc68495c21/payload-pre-processing/0.log" Jul 14 18:56:10.547549 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:10.547517 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-gfmmb_29b98be0-12b3-4486-a0ce-ca7627424ecb/payload-processing/0.log" Jul 14 18:56:11.272196 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:11.272166 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55b7dd5b8c-5w45m_ade5c338-5d02-4aec-be76-f71fb89e6fcc/manager/0.log" Jul 14 18:56:11.331995 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:11.331968 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-t8bxs_7bb08805-5a28-4db4-b5f4-cfd66a1ee67d/openshift-lws-operator/0.log" Jul 14 18:56:11.923518 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:11.923483 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565f78b694-fwm48_2f4c8035-e58d-45d4-b82e-513983279c85/kube-auth-proxy/0.log" Jul 14 18:56:12.018286 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:12.018252 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-4dk9s_79ccee40-6189-4489-8087-a3dc68495c21/payload-pre-processing/0.log" Jul 14 18:56:12.041570 ip-10-0-131-214 kubenswrapper[2567]: I0714 18:56:12.041540 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-gfmmb_29b98be0-12b3-4486-a0ce-ca7627424ecb/payload-processing/0.log"