Jul 06 19:27:22.233169 ip-10-0-130-143 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.613203 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619940 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619956 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619960 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619963 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619965 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619968 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619971 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.685958 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619974 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619977 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619980 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619982 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619985 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619988 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619991 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619994 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.619996 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620000 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620003 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620006 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620009 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620012 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620018 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620021 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620025 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620028 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620031 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620033 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.687235 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620036 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620039 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620041 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620044 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620047 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620050 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620052 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620055 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620058 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620061 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620063 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620068 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620071 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620074 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620077 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620079 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620082 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620084 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620087 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.688771 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620089 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.688196 ip-10-0-130-143 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620092 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620095 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620098 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620101 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620104 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620108 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620111 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620114 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620118 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620121 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620124 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620128 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620130 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620133 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620136 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620138 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620142 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620144 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620147 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.690795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620150 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620153 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620155 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620158 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620163 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620166 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620170 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620172 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620175 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620178 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620180 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620183 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620186 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620188 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620191 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620194 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620197 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620199 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620202 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.693472 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620205 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620207 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620210 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620213 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620215 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620218 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620220 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620223 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620225 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620228 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620231 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620234 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620237 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620946 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620953 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620956 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620959 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620962 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620964 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620967 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.694131 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620970 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620973 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620976 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620978 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620982 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620986 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620989 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620993 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.620997 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621000 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621004 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621007 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621010 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621013 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621016 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621018 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621021 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621023 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621026 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.694800 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621028 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621031 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621034 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621038 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621041 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621043 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621046 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621048 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621051 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621054 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621056 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621059 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621061 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621064 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621067 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621069 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621072 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621074 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621077 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621079 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.695728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621082 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621085 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621088 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621092 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621096 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621099 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621102 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621105 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621108 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621111 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621114 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621116 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621119 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621121 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621124 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621127 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621130 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621132 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621135 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.696645 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621137 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621140 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621142 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621145 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621147 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621150 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621152 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621155 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621158 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621160 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621163 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621165 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621168 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621173 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621176 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621178 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621181 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621184 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621187 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.697385 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621190 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621192 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621195 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621198 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621200 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621203 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621206 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621209 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621211 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621214 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621217 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621219 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621222 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621224 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621227 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621625 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621630 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621633 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621636 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621638 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.698180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621641 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621643 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621647 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621649 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621652 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621654 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621659 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621662 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621665 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621667 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621670 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621674 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621677 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621680 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621682 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621685 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621688 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621690 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621693 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621696 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.699041 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621699 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621702 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621705 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621707 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621710 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621713 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621716 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621718 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621721 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621724 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621726 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621729 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621732 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621734 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621737 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621739 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621742 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621745 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621748 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621751 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.699870 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621754 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621756 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621759 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621761 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621766 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621768 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621771 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621774 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621777 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621780 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621782 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621785 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621787 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621790 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621792 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621795 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621797 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621800 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621803 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.700690 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621806 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621811 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621814 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621817 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621820 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621823 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621825 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621828 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621830 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621833 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621836 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621839 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621842 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621844 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621847 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621850 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621852 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621855 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621862 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.701463 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621865 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621869 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621872 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621876 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621879 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621882 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621885 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621888 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621891 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621893 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621912 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621916 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621919 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621922 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621924 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.621927 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.621995 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622002 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622008 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622013 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:27:22.702306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622017 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622021 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622025 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622029 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622034 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622037 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622041 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622044 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622047 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622050 2577 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622053 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622058 2577 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622062 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622065 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622069 2577 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622072 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622075 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622078 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622082 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622086 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622089 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622092 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622095 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622098 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:27:22.702924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622101 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622104 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622107 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622110 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622117 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622120 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622123 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622126 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622129 2577 flags.go:64] FLAG: --enable-server="true" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622131 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622136 2577 flags.go:64] FLAG: --event-burst="100" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622139 2577 flags.go:64] FLAG: --event-qps="50" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622143 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622146 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622149 2577 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622153 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622156 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622159 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622162 2577 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622166 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622169 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622173 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622176 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622179 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622182 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:27:22.703672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622185 2577 flags.go:64] FLAG: --feature-gates="" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622189 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622192 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622195 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622198 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622202 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622205 2577 flags.go:64] FLAG: --help="false" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622208 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622211 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622214 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622216 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622220 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622224 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622227 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622230 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622233 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622235 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622238 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622242 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622246 2577 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622248 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622251 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622255 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622257 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:27:22.704656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622260 2577 flags.go:64] FLAG: --lock-file="" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622263 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622267 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622270 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622276 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622279 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622282 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622285 2577 flags.go:64] FLAG: --logging-format="text" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622288 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622291 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622294 2577 flags.go:64] FLAG: --manifest-url="" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622297 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622301 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622304 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622308 2577 flags.go:64] FLAG: --max-pods="110" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622311 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622314 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622317 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622320 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622323 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622326 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622329 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622336 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622339 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622342 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:27:22.705647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622345 2577 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622348 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622354 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622359 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622362 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622365 2577 flags.go:64] FLAG: --port="10250" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622368 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622371 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5229a5e5ef8b141" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622374 2577 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622378 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622381 2577 flags.go:64] FLAG: --register-node="true" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622384 2577 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622388 2577 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622391 2577 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622394 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622397 2577 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622400 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622403 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622406 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622410 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622412 2577 flags.go:64] FLAG: --runonce="false" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622415 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622418 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622421 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622424 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:27:22.706429 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622427 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622430 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622433 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622436 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622439 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622442 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622444 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622447 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622450 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622453 2577 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622457 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622463 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622465 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622469 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622472 2577 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622475 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622479 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622482 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622485 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622488 2577 flags.go:64] FLAG: --v="2" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622492 2577 flags.go:64] FLAG: --version="false" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622496 2577 flags.go:64] FLAG: --vmodule="" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622500 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.622503 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622610 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.707452 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622614 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622617 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622619 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622622 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622625 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622628 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622631 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622634 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622636 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622639 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622642 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622644 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622647 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622651 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622654 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622657 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622660 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622664 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622668 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.708394 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622672 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622676 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622679 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622682 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622687 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622690 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622693 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622696 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622699 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622702 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622705 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622708 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622710 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622713 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622715 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622718 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622721 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622723 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622726 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622729 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.709170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622732 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622734 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622737 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622739 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622742 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622744 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622747 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622750 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622752 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622755 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622759 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622761 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622764 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622767 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622770 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622772 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622776 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622779 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622781 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622784 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.709993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622786 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622789 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622792 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622795 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622797 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622800 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622802 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622805 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622808 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622811 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622814 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622816 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622819 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622821 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622824 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622826 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622829 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622832 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622834 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622837 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.710608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622839 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622842 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622847 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622850 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622853 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622856 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622859 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622861 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622865 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622868 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622871 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622873 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622876 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622878 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622881 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622883 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622887 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622889 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.711179 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.622892 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.623432 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.629751 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.629766 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.629782 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.629787 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629843 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629848 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629852 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629855 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629858 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629861 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629864 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629867 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.711663 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629869 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629872 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629874 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629877 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629880 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629883 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629885 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629888 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629891 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629894 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629914 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629917 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629919 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629922 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629925 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629928 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629930 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629934 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629938 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629941 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629943 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.712058 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629947 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629949 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629952 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629955 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629957 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629960 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629963 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629965 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629968 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629971 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629973 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629976 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629978 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629981 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629983 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629986 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629988 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629991 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629994 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629997 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.712648 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.629999 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630002 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630004 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630007 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630010 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630013 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630015 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630018 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630021 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630024 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630028 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630032 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630035 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630038 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630041 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630043 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630046 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630048 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630051 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630053 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.713170 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630056 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630058 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630061 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630063 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630066 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630068 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630071 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630074 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630076 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630079 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630082 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630086 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630089 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630092 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630095 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630097 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630100 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630103 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630105 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630108 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.713673 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630110 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630113 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630117 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630121 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630124 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630127 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630132 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630135 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630138 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630141 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.630146 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630265 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630270 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630273 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:27:22.714174 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630276 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630278 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630281 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630284 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630286 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630289 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630291 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630294 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630296 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630301 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630304 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630306 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630309 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630311 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630314 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630317 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630319 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630322 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630325 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:27:22.714528 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630328 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630330 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630333 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630336 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630339 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630341 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630345 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630347 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630351 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630356 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630358 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630361 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630364 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630367 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630370 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630372 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630375 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630378 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630380 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:27:22.714995 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630383 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630385 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630388 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630392 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630395 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630397 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630400 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630403 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630405 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630408 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630411 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630413 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630416 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630421 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630424 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630426 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630429 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630432 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630434 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630437 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:27:22.715459 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630442 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630445 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630448 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630451 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630453 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630456 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630458 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630461 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630464 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630466 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630469 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630472 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630476 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630479 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630482 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630487 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630489 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630493 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630497 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630500 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:27:22.715965 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630502 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630505 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630508 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630510 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630512 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630517 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630520 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630522 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630525 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630528 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630530 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630533 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630536 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630539 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630542 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630544 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630547 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:27:22.716614 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:22.630550 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.630554 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.630648 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.632479 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.634087 2577 server.go:1019] "Starting client certificate rotation" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.634180 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.634211 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.636413 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.638395 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.652247 2577 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.655252 2577 log.go:25] "Validated CRI v1 image API" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.656050 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:27:22.717081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.660965 2577 fs.go:135] Filesystem UUIDs: map[1ad7b572-3224-4059-8dce-9963ca433367:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ce4660c-652f-490b-be35-a9e6717d7308:/dev/nvme0n1p4] Jul 06 19:27:22.717489 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.660978 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.666155 2577 manager.go:216] Machine: {Timestamp:2026-07-06 19:27:22.664910639 +0000 UTC m=+0.318478230 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087189 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2420497a318b09fceb329e4e3a3874 SystemUUID:ec242049-7a31-8b09-fceb-329e4e3a3874 BootID:a46a3d82-76dd-443a-bdd4-19fa8373e01f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:6c:b3:92:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:6c:b3:92:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:8b:81:7b:69:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.666250 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.666321 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.667863 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.667885 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.669173 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.669182 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:27:22.717525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.669190 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.669322 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.669717 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.670838 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.673962 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674069 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674095 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674103 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674113 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674121 2577 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674130 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674423 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.674439 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.676999 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.678916 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.678957 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679957 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679969 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679975 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679981 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679989 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.679995 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680003 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680008 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680015 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680021 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680029 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680034 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680774 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.680789 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.683753 2577 server.go:1268] "Started kubelet" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.683853 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.683910 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.683970 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.687757 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.688023 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.689258 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.689879 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.690091 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:27:22.717772 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.690177 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.692541 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-143.ec2.internal.18bfc9f796a85a15 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-143.ec2.internal,UID:ip-10-0-130-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-143.ec2.internal,},FirstTimestamp:2026-07-06 19:27:22.683726357 +0000 UTC m=+0.337293949,LastTimestamp:2026-07-06 19:27:22.683726357 +0000 UTC m=+0.337293949,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-143.ec2.internal,}" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.694079 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wp5tk" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.694699 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.694708 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.694814 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.695236 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.695283 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.695294 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.695352 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.695363 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698480 2577 factory.go:153] Registering CRI-O factory Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698493 2577 factory.go:223] Registration of the crio container factory successfully Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698540 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698549 2577 factory.go:55] Registering systemd factory Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698556 2577 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698577 2577 factory.go:103] Registering Raw factory Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.698591 2577 manager.go:1196] Started watching for new ooms in manager Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.699431 2577 manager.go:349] Starting recovery of all containers Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.703622 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wp5tk" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.705275 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.705589 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.709692 2577 manager.go:354] Recovery completed Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.714116 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716118 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716144 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716153 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716591 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716602 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716618 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.716660 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717394 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.718933 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717416 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717428 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717625 2577 policy_none.go:49] "None policy: Start" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717642 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.717654 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.718058 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-143.ec2.internal.18bfc9f79896d002 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-143.ec2.internal,UID:ip-10-0-130-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-143.ec2.internal,},FirstTimestamp:2026-07-06 19:27:22.71613133 +0000 UTC m=+0.369698919,LastTimestamp:2026-07-06 19:27:22.71613133 +0000 UTC m=+0.369698919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-143.ec2.internal,}" Jul 06 19:27:22.720014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.718706 2577 policy_none.go:47] "Start" Jul 06 19:27:22.744431 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744410 2577 manager.go:337] "Starting Device Plugin manager" Jul 06 19:27:22.744503 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.744449 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:27:22.744503 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744460 2577 server.go:87] "Starting device plugin registration server" Jul 06 19:27:22.744647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744635 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:27:22.744692 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744649 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:27:22.744733 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744720 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:27:22.744786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744775 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:27:22.744828 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.744788 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:27:22.745255 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.745234 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:27:22.745335 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.745284 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:22.801890 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.801871 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:27:22.804333 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.803004 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:27:22.804333 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.803044 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:27:22.804333 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.803064 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:27:22.804333 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.803103 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:27:22.806008 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.805994 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:22.845301 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.845243 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.846052 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.846039 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.846118 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.846065 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.846118 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.846080 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.846118 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.846104 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.853059 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.853046 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.853098 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.853065 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-143.ec2.internal\": node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:22.870474 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.870459 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:22.903745 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.903725 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal"] Jul 06 19:27:22.903805 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.903775 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.905101 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.905085 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.905164 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.905114 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.905164 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.905124 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.906071 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906060 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.906188 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.906231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906202 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.906722 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906708 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.906774 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906736 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.906774 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.906746 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.907226 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.907214 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.907265 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.907239 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.907265 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.907251 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.907749 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.907736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.907785 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.907759 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:27:22.908365 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.908347 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:27:22.908432 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.908372 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:27:22.908432 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.908385 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:27:22.933523 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.933400 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-143.ec2.internal\" not found" node="ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.937506 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.937491 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-143.ec2.internal\" not found" node="ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.971078 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:22.971060 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:22.997113 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.997094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.997174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.997116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa627ad35b77ae9bde45172877976b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-143.ec2.internal\" (UID: \"aa627ad35b77ae9bde45172877976b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:22.997174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:22.997133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.071947 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.071928 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:23.098288 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.098288 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.098288 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa627ad35b77ae9bde45172877976b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-143.ec2.internal\" (UID: \"aa627ad35b77ae9bde45172877976b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.098411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa627ad35b77ae9bde45172877976b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-143.ec2.internal\" (UID: \"aa627ad35b77ae9bde45172877976b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.098411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.098411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.098338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/944a10815a5caee6dab36f61ae5d0e90-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal\" (UID: \"944a10815a5caee6dab36f61ae5d0e90\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.172515 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.172496 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:23.235979 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.235962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.240506 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.240489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.273122 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.273102 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:23.373505 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.373463 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:23.473846 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.473831 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-143.ec2.internal\" not found" Jul 06 19:27:23.554096 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.554078 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:23.595975 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.595956 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.607492 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.607479 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:27:23.609120 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.609105 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" Jul 06 19:27:23.618204 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.618122 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:27:23.633744 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.633698 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:27:23.633829 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.633809 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:27:23.633879 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.633809 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:27:23.674952 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.674934 2577 apiserver.go:52] "Watching apiserver" Jul 06 19:27:23.692882 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.692866 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:27:23.693206 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.693153 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hbm82","kube-system/global-pull-secret-syncer-65kqw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd","openshift-cluster-node-tuning-operator/tuned-2kvsm","openshift-dns/node-resolver-xkrj6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal","openshift-multus/multus-4pxqm","openshift-multus/network-metrics-daemon-xwq6w","openshift-network-diagnostics/network-check-target-jqwdl","kube-system/konnectivity-agent-7qgxs","kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal","openshift-image-registry/node-ca-7z4x5","openshift-multus/multus-additional-cni-plugins-8cn5l","openshift-network-operator/iptables-alerter-qnw2t"] Jul 06 19:27:23.694960 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.694945 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:27:23.695414 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.695401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.697920 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.697691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.698388 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.698339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.698494 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.698448 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h99cc\"" Jul 06 19:27:23.698564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.698543 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:27:23.698617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.698550 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.698670 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.698652 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:27:23.700233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.700372 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g28j\" (UniqueName: \"kubernetes.io/projected/85fbf08f-f42b-4610-bbf4-2808231b4550-kube-api-access-4g28j\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-system-cni-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.700441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-cnibin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-os-release\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700251 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-netns\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428vd\" (UniqueName: \"kubernetes.io/projected/a8830363-3b24-4e4b-94e9-635775bc5552-kube-api-access-428vd\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-cni-binary-copy\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-bin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700601 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bbbsc\"" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-multus\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-kubelet\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.700687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-daemon-config\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-etc-kubernetes\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-cnibin\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-k8s-cni-cncf-io\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-conf-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-multus-certs\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-os-release\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-system-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-socket-dir-parent\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.700991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-hostroot\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.701157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.701126 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:27:23.702212 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.702199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.702605 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.702583 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:27:23.702684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.702662 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.702684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.702672 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9rrt\"" Jul 06 19:27:23.702750 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.702728 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.704129 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.704113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.704508 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.704494 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.704625 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.704600 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.704770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.704759 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cck9p\"" Jul 06 19:27:23.706158 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.706139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.706338 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.706319 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.706559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.706540 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sq7z5\"" Jul 06 19:27:23.706626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.706607 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.707181 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.707168 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:27:23.707406 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.707377 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:22:22 +0000 UTC" deadline="2028-04-11 22:35:33.073173492 +0000 UTC" Jul 06 19:27:23.707406 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.707406 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15483h8m9.3657695s" Jul 06 19:27:23.708201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.708187 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:23.708283 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.708234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:23.708740 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.708721 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.708854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.708721 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:27:23.708854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.708812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9hz74\"" Jul 06 19:27:23.708854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.708814 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.710646 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.710631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.712809 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.712788 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.712933 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.712843 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:23.713086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:27:23.713086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713083 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6s68c\"" Jul 06 19:27:23.713464 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713450 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:27:23.713547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713495 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.713547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713516 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.713547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.713450 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:27:23.714415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.714402 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:27:23.714857 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.714847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.717077 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.717064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.717140 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.717122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:23.717186 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.717157 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:27:23.717841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.717823 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:27:23.718588 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.718569 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gwvng\"" Jul 06 19:27:23.719043 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.719027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.721389 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.721371 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:27:23.721459 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.721445 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qvtqt\"" Jul 06 19:27:23.721693 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.721682 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:27:23.722619 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.722600 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:27:23.723767 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:23.723738 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa627ad35b77ae9bde45172877976b6e.slice/crio-fa98fabc270e49b6dbd50f3648994d1e2bace232727fa68e4fd052e98dd2a3e4 WatchSource:0}: Error finding container fa98fabc270e49b6dbd50f3648994d1e2bace232727fa68e4fd052e98dd2a3e4: Status 404 returned error can't find the container with id fa98fabc270e49b6dbd50f3648994d1e2bace232727fa68e4fd052e98dd2a3e4 Jul 06 19:27:23.724195 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:23.724175 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944a10815a5caee6dab36f61ae5d0e90.slice/crio-9e94780178c603401fd2eeff1ad21a3e92dc25d87d31397af6c273589af97055 WatchSource:0}: Error finding container 9e94780178c603401fd2eeff1ad21a3e92dc25d87d31397af6c273589af97055: Status 404 returned error can't find the container with id 9e94780178c603401fd2eeff1ad21a3e92dc25d87d31397af6c273589af97055 Jul 06 19:27:23.729488 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.729402 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:27:23.729621 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.729606 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zn4fj" Jul 06 19:27:23.737859 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.737845 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zn4fj" Jul 06 19:27:23.796043 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.796026 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:27:23.801616 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.801664 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.801664 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-device-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.801664 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-modprobe-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-env-overrides\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-k8s-cni-cncf-io\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-slash\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-script-lib\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.801782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801766 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-k8s-cni-cncf-io\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g28j\" (UniqueName: \"kubernetes.io/projected/85fbf08f-f42b-4610-bbf4-2808231b4550-kube-api-access-4g28j\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/af96c196-be64-4514-826b-4db41c7f1eca-kube-api-access-kd525\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-bin\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-netns\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.801994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801958 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-netns\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.801996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428vd\" (UniqueName: \"kubernetes.io/projected/a8830363-3b24-4e4b-94e9-635775bc5552-kube-api-access-428vd\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca5c635-0f15-4354-a416-513cbc89b7bb-serviceca\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh945\" (UniqueName: \"kubernetes.io/projected/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kube-api-access-sh945\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-sys\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-var-lib-kubelet\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-config\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.802322 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802322 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-cni-binary-copy\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802322 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-multus\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802322 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-systemd\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-host\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802310 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-multus\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-tmp-dir\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccfd5d69-d911-484e-8b09-bcd602660c1a-host-slash\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.802438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-conf-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-multus-certs\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-conf-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-run-multus-certs\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-log-socket\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8h5\" (UniqueName: \"kubernetes.io/projected/4661f471-2b64-4e51-9361-f90b1dcc9910-kube-api-access-7g8h5\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-system-cni-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62nq8\" (UniqueName: \"kubernetes.io/projected/7ca5c635-0f15-4354-a416-513cbc89b7bb-kube-api-access-62nq8\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802589 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-system-cni-dir\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-conf\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.802617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-run\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccfd5d69-d911-484e-8b09-bcd602660c1a-iptables-alerter-script\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-node-log\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802670 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-cni-binary-copy\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-cnibin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-etc-kubernetes\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-cnibin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gptgw\" (UniqueName: \"kubernetes.io/projected/e5d39682-b7a1-4d77-b913-3475037e2d43-kube-api-access-gptgw\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-etc-kubernetes\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.803153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-etc-tuned\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802869 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-tmp\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-netns\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5d39682-b7a1-4d77-b913-3475037e2d43-ovn-node-metrics-cert\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.802982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-system-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803036 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-system-cni-dir\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-hostroot\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803078 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-hostroot\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-kubernetes\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-lib-modules\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ddf894a6-f277-4b6b-8306-ab44bdce81bf-agent-certs\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.803738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-hosts-file\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-os-release\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-dbus\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-os-release\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-bin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803330 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-cni-bin\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-daemon-config\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-cnibin\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca5c635-0f15-4354-a416-513cbc89b7bb-host\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysconfig\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803441 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-cnibin\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rk4\" (UniqueName: \"kubernetes.io/projected/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-kube-api-access-28rk4\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-kubelet-config\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-systemd-units\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-systemd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-ovn\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-socket-dir-parent\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-kubelet\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-netd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803641 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8830363-3b24-4e4b-94e9-635775bc5552-cni-binary-copy\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ddf894a6-f277-4b6b-8306-ab44bdce81bf-konnectivity-ca\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9f4\" (UniqueName: \"kubernetes.io/projected/ccfd5d69-d911-484e-8b09-bcd602660c1a-kube-api-access-2s9f4\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-socket-dir-parent\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-var-lib-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.804782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-etc-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.805307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-kubelet\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.805307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85fbf08f-f42b-4610-bbf4-2808231b4550-multus-daemon-config\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.805307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-os-release\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.805307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85fbf08f-f42b-4610-bbf4-2808231b4550-host-var-lib-kubelet\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.805307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.803858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8830363-3b24-4e4b-94e9-635775bc5552-os-release\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.805549 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.805508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" event={"ID":"aa627ad35b77ae9bde45172877976b6e","Type":"ContainerStarted","Data":"fa98fabc270e49b6dbd50f3648994d1e2bace232727fa68e4fd052e98dd2a3e4"} Jul 06 19:27:23.806382 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.806365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" event={"ID":"944a10815a5caee6dab36f61ae5d0e90","Type":"ContainerStarted","Data":"9e94780178c603401fd2eeff1ad21a3e92dc25d87d31397af6c273589af97055"} Jul 06 19:27:23.808612 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.808590 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:27:23.811302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.811278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g28j\" (UniqueName: \"kubernetes.io/projected/85fbf08f-f42b-4610-bbf4-2808231b4550-kube-api-access-4g28j\") pod \"multus-4pxqm\" (UID: \"85fbf08f-f42b-4610-bbf4-2808231b4550\") " pod="openshift-multus/multus-4pxqm" Jul 06 19:27:23.811369 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.811275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428vd\" (UniqueName: \"kubernetes.io/projected/a8830363-3b24-4e4b-94e9-635775bc5552-kube-api-access-428vd\") pod \"multus-additional-cni-plugins-8cn5l\" (UID: \"a8830363-3b24-4e4b-94e9-635775bc5552\") " pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:23.828644 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.828631 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:23.904308 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-slash\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904308 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-script-lib\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904308 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/af96c196-be64-4514-826b-4db41c7f1eca-kube-api-access-kd525\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904339 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-slash\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-bin\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.904421 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca5c635-0f15-4354-a416-513cbc89b7bb-serviceca\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh945\" (UniqueName: \"kubernetes.io/projected/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kube-api-access-sh945\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.904486 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:24.404463891 +0000 UTC m=+2.058031491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:23.904514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-sys\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-var-lib-kubelet\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904550 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-sys\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-config\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-bin\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-systemd\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-host\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-tmp-dir\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccfd5d69-d911-484e-8b09-bcd602660c1a-host-slash\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904658 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-systemd\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-log-socket\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-var-lib-kubelet\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-host\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8h5\" (UniqueName: \"kubernetes.io/projected/4661f471-2b64-4e51-9361-f90b1dcc9910-kube-api-access-7g8h5\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccfd5d69-d911-484e-8b09-bcd602660c1a-host-slash\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.905046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62nq8\" (UniqueName: \"kubernetes.io/projected/7ca5c635-0f15-4354-a416-513cbc89b7bb-kube-api-access-62nq8\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-log-socket\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-conf\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca5c635-0f15-4354-a416-513cbc89b7bb-serviceca\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-run\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccfd5d69-d911-484e-8b09-bcd602660c1a-iptables-alerter-script\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-node-log\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904891 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-run\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-conf\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-script-lib\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.905728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.904981 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-node-log\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905011 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-tmp-dir\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gptgw\" (UniqueName: \"kubernetes.io/projected/e5d39682-b7a1-4d77-b913-3475037e2d43-kube-api-access-gptgw\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysctl-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-etc-tuned\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-tmp\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-ovnkube-config\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-netns\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5d39682-b7a1-4d77-b913-3475037e2d43-ovn-node-metrics-cert\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-kubernetes\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-netns\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-lib-modules\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.905217 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ddf894a6-f277-4b6b-8306-ab44bdce81bf-agent-certs\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.905293 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:24.405277054 +0000 UTC m=+2.058844647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-kubernetes\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-hosts-file\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-dbus\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905386 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-lib-modules\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca5c635-0f15-4354-a416-513cbc89b7bb-host\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysconfig\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905433 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca5c635-0f15-4354-a416-513cbc89b7bb-host\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rk4\" (UniqueName: \"kubernetes.io/projected/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-kube-api-access-28rk4\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-kubelet-config\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.906870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905479 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-systemd-units\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-systemd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-ovn\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-hosts-file\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-sysconfig\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905621 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-kubelet\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-systemd-units\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905650 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-kubelet\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905650 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-kubelet-config\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-netd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905682 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-host-cni-netd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-systemd\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-run-ovn\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ddf894a6-f277-4b6b-8306-ab44bdce81bf-konnectivity-ca\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9f4\" (UniqueName: \"kubernetes.io/projected/ccfd5d69-d911-484e-8b09-bcd602660c1a-kube-api-access-2s9f4\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d11926aa-a0cc-40d1-9c68-17a44704d1b6-dbus\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-var-lib-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-var-lib-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-etc-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-device-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.905982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-modprobe-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-env-overrides\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.907680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccfd5d69-d911-484e-8b09-bcd602660c1a-iptables-alerter-script\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906005 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5d39682-b7a1-4d77-b913-3475037e2d43-etc-openvswitch\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-device-dir\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af96c196-be64-4514-826b-4db41c7f1eca-etc-modprobe-d\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906156 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5d39682-b7a1-4d77-b913-3475037e2d43-env-overrides\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.906587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ddf894a6-f277-4b6b-8306-ab44bdce81bf-konnectivity-ca\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.907439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-tmp\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.907448 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af96c196-be64-4514-826b-4db41c7f1eca-etc-tuned\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.907519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5d39682-b7a1-4d77-b913-3475037e2d43-ovn-node-metrics-cert\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.908061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.907719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ddf894a6-f277-4b6b-8306-ab44bdce81bf-agent-certs\") pod \"konnectivity-agent-7qgxs\" (UID: \"ddf894a6-f277-4b6b-8306-ab44bdce81bf\") " pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:23.920847 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.920821 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8h5\" (UniqueName: \"kubernetes.io/projected/4661f471-2b64-4e51-9361-f90b1dcc9910-kube-api-access-7g8h5\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:23.921336 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.921321 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gptgw\" (UniqueName: \"kubernetes.io/projected/e5d39682-b7a1-4d77-b913-3475037e2d43-kube-api-access-gptgw\") pod \"ovnkube-node-hbm82\" (UID: \"e5d39682-b7a1-4d77-b913-3475037e2d43\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:23.922238 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.922224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9f4\" (UniqueName: \"kubernetes.io/projected/ccfd5d69-d911-484e-8b09-bcd602660c1a-kube-api-access-2s9f4\") pod \"iptables-alerter-qnw2t\" (UID: \"ccfd5d69-d911-484e-8b09-bcd602660c1a\") " pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:23.924121 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.924076 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rk4\" (UniqueName: \"kubernetes.io/projected/a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a-kube-api-access-28rk4\") pod \"node-resolver-xkrj6\" (UID: \"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a\") " pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:23.924195 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.924188 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:23.924232 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.924199 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:23.924232 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.924227 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:23.924302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.924259 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh945\" (UniqueName: \"kubernetes.io/projected/327f5a30-a616-42ab-a70c-0bb02cfdd2d2-kube-api-access-sh945\") pod \"aws-ebs-csi-driver-node-dzcfd\" (UID: \"327f5a30-a616-42ab-a70c-0bb02cfdd2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:23.924302 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:23.924268 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:24.424256786 +0000 UTC m=+2.077824361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:23.925960 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.925942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62nq8\" (UniqueName: \"kubernetes.io/projected/7ca5c635-0f15-4354-a416-513cbc89b7bb-kube-api-access-62nq8\") pod \"node-ca-7z4x5\" (UID: \"7ca5c635-0f15-4354-a416-513cbc89b7bb\") " pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:23.927571 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:23.927554 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/af96c196-be64-4514-826b-4db41c7f1eca-kube-api-access-kd525\") pod \"tuned-2kvsm\" (UID: \"af96c196-be64-4514-826b-4db41c7f1eca\") " pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:24.026511 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.026489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pxqm" Jul 06 19:27:24.033653 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.033633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" Jul 06 19:27:24.035460 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.035438 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fbf08f_f42b_4610_bbf4_2808231b4550.slice/crio-a025db79c64ec99c7d4f217aefeb9c74e27551daa0c7d26324c407488935b611 WatchSource:0}: Error finding container a025db79c64ec99c7d4f217aefeb9c74e27551daa0c7d26324c407488935b611: Status 404 returned error can't find the container with id a025db79c64ec99c7d4f217aefeb9c74e27551daa0c7d26324c407488935b611 Jul 06 19:27:24.040360 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.040346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" Jul 06 19:27:24.040993 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.040973 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8830363_3b24_4e4b_94e9_635775bc5552.slice/crio-6a1365170917cbe42ce9bdffcd2f915768563dcd24947937889bd29d7232ac9b WatchSource:0}: Error finding container 6a1365170917cbe42ce9bdffcd2f915768563dcd24947937889bd29d7232ac9b: Status 404 returned error can't find the container with id 6a1365170917cbe42ce9bdffcd2f915768563dcd24947937889bd29d7232ac9b Jul 06 19:27:24.044166 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.044153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" Jul 06 19:27:24.047210 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.047195 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327f5a30_a616_42ab_a70c_0bb02cfdd2d2.slice/crio-b282250dff92467c9d33fbd6fe116dfd55eb5286633bc656b3349fc19ad84c1f WatchSource:0}: Error finding container b282250dff92467c9d33fbd6fe116dfd55eb5286633bc656b3349fc19ad84c1f: Status 404 returned error can't find the container with id b282250dff92467c9d33fbd6fe116dfd55eb5286633bc656b3349fc19ad84c1f Jul 06 19:27:24.048565 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.048549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xkrj6" Jul 06 19:27:24.051782 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.051764 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf96c196_be64_4514_826b_4db41c7f1eca.slice/crio-f41a34a97e817c385b53f855b0daf8a10a5b51811177d82aa3ae2ec484f0f11e WatchSource:0}: Error finding container f41a34a97e817c385b53f855b0daf8a10a5b51811177d82aa3ae2ec484f0f11e: Status 404 returned error can't find the container with id f41a34a97e817c385b53f855b0daf8a10a5b51811177d82aa3ae2ec484f0f11e Jul 06 19:27:24.054431 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.054420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnw2t" Jul 06 19:27:24.056368 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.056336 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61d4a93_c673_46d3_a5c1_bc5dd06c8c7a.slice/crio-fb9ece1c34f584329a35156208ba8272856af69fdc15df2c32ef2b319f2bf4f6 WatchSource:0}: Error finding container fb9ece1c34f584329a35156208ba8272856af69fdc15df2c32ef2b319f2bf4f6: Status 404 returned error can't find the container with id fb9ece1c34f584329a35156208ba8272856af69fdc15df2c32ef2b319f2bf4f6 Jul 06 19:27:24.059002 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.058955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:24.062805 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.062785 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfd5d69_d911_484e_8b09_bcd602660c1a.slice/crio-f8a2673327b092ccf116e62709d9ec130159b53b32021a29a9c323b783e2966c WatchSource:0}: Error finding container f8a2673327b092ccf116e62709d9ec130159b53b32021a29a9c323b783e2966c: Status 404 returned error can't find the container with id f8a2673327b092ccf116e62709d9ec130159b53b32021a29a9c323b783e2966c Jul 06 19:27:24.063486 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.063467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:24.068014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.067997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7z4x5" Jul 06 19:27:24.068231 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.068213 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d39682_b7a1_4d77_b913_3475037e2d43.slice/crio-8c0516d7a661031fc98c43db6f6522e20ef78300c874dac518f5ecbe6d8af721 WatchSource:0}: Error finding container 8c0516d7a661031fc98c43db6f6522e20ef78300c874dac518f5ecbe6d8af721: Status 404 returned error can't find the container with id 8c0516d7a661031fc98c43db6f6522e20ef78300c874dac518f5ecbe6d8af721 Jul 06 19:27:24.072062 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.072044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf894a6_f277_4b6b_8306_ab44bdce81bf.slice/crio-75195c2b3adfd66ad45b8be7cedcaef6087491ed88f4012d074538f1abb20504 WatchSource:0}: Error finding container 75195c2b3adfd66ad45b8be7cedcaef6087491ed88f4012d074538f1abb20504: Status 404 returned error can't find the container with id 75195c2b3adfd66ad45b8be7cedcaef6087491ed88f4012d074538f1abb20504 Jul 06 19:27:24.077031 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:24.077008 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca5c635_0f15_4354_a416_513cbc89b7bb.slice/crio-92a9c662a3f6229b30afc2b776951b5df15ef08236c6bd76627fe62160727f4f WatchSource:0}: Error finding container 92a9c662a3f6229b30afc2b776951b5df15ef08236c6bd76627fe62160727f4f: Status 404 returned error can't find the container with id 92a9c662a3f6229b30afc2b776951b5df15ef08236c6bd76627fe62160727f4f Jul 06 19:27:24.246117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.246051 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:24.409020 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.408983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:24.409186 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.409048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:24.409186 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.409146 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:24.409186 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.409178 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:24.409405 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.409220 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:25.409198014 +0000 UTC m=+3.062765593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:24.409405 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.409254 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:25.409230969 +0000 UTC m=+3.062798549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:24.509936 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.509842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:24.510087 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.510030 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:24.510087 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.510048 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:24.510087 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.510060 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:24.510279 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.510127 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:25.510109063 +0000 UTC m=+3.163676657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:24.741779 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.741712 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:22:23 +0000 UTC" deadline="2028-01-01 12:40:28.585841831 +0000 UTC" Jul 06 19:27:24.741779 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.741746 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13049h13m3.84410087s" Jul 06 19:27:24.805171 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.805105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:24.805320 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:24.805223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:24.817576 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.817491 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:24.847183 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.847152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pxqm" event={"ID":"85fbf08f-f42b-4610-bbf4-2808231b4550","Type":"ContainerStarted","Data":"a025db79c64ec99c7d4f217aefeb9c74e27551daa0c7d26324c407488935b611"} Jul 06 19:27:24.863341 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.863307 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7z4x5" event={"ID":"7ca5c635-0f15-4354-a416-513cbc89b7bb","Type":"ContainerStarted","Data":"92a9c662a3f6229b30afc2b776951b5df15ef08236c6bd76627fe62160727f4f"} Jul 06 19:27:24.891102 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.891070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7qgxs" event={"ID":"ddf894a6-f277-4b6b-8306-ab44bdce81bf","Type":"ContainerStarted","Data":"75195c2b3adfd66ad45b8be7cedcaef6087491ed88f4012d074538f1abb20504"} Jul 06 19:27:24.906207 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.906180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"8c0516d7a661031fc98c43db6f6522e20ef78300c874dac518f5ecbe6d8af721"} Jul 06 19:27:24.911358 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.911296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xkrj6" event={"ID":"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a","Type":"ContainerStarted","Data":"fb9ece1c34f584329a35156208ba8272856af69fdc15df2c32ef2b319f2bf4f6"} Jul 06 19:27:24.931716 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.931644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" event={"ID":"af96c196-be64-4514-826b-4db41c7f1eca","Type":"ContainerStarted","Data":"f41a34a97e817c385b53f855b0daf8a10a5b51811177d82aa3ae2ec484f0f11e"} Jul 06 19:27:24.950551 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.950527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerStarted","Data":"6a1365170917cbe42ce9bdffcd2f915768563dcd24947937889bd29d7232ac9b"} Jul 06 19:27:24.971470 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.971443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnw2t" event={"ID":"ccfd5d69-d911-484e-8b09-bcd602660c1a","Type":"ContainerStarted","Data":"f8a2673327b092ccf116e62709d9ec130159b53b32021a29a9c323b783e2966c"} Jul 06 19:27:24.982093 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:24.982068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" event={"ID":"327f5a30-a616-42ab-a70c-0bb02cfdd2d2","Type":"ContainerStarted","Data":"b282250dff92467c9d33fbd6fe116dfd55eb5286633bc656b3349fc19ad84c1f"} Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.417875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.417955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.418088 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.418150 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:27.418130926 +0000 UTC m=+5.071698508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.418523 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:25.419938 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.418572 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:27.418557178 +0000 UTC m=+5.072124757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:25.519723 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.519147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:25.519723 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.519293 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:25.519723 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.519314 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:25.519723 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.519326 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:25.519723 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.519382 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:27.519364875 +0000 UTC m=+5.172932471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:25.742424 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.742317 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:22:23 +0000 UTC" deadline="2028-03-05 12:36:35.84146965 +0000 UTC" Jul 06 19:27:25.742424 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.742375 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14585h9m10.099098737s" Jul 06 19:27:25.804336 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.803623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:25.804336 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.803744 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:25.804336 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:25.804136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:25.804336 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:25.804286 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:26.602114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:26.602081 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:27:26.805196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:26.804620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:26.805196 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:26.804753 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:27.435124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:27.435052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:27.435273 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:27.435130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:27.435336 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.435282 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:27.435391 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.435349 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:31.435328995 +0000 UTC m=+9.088896589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:27.435745 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.435726 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:27.435837 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.435786 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:31.435768897 +0000 UTC m=+9.089336476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:27.535911 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:27.535861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:27.536110 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.536092 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:27.536199 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.536121 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:27.536199 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.536134 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:27.536199 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.536192 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:31.536172769 +0000 UTC m=+9.189740348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:27.803943 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:27.803831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:27.804103 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.803966 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:27.804103 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:27.803831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:27.804213 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:27.804124 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:28.803558 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:28.803527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:28.804154 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:28.803658 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:29.803562 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:29.803418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:29.803562 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:29.803557 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:29.804091 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:29.803920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:29.804091 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:29.804013 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:30.803805 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:30.803613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:30.803805 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:30.803738 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:31.466087 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:31.466053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:31.466259 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:31.466119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:31.466259 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.466229 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:31.466259 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.466230 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:31.466424 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.466291 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:39.466275998 +0000 UTC m=+17.119843575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:31.466424 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.466313 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:39.466301938 +0000 UTC m=+17.119869520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:31.567512 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:31.567473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:31.567680 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.567627 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:31.567680 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.567644 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:31.567680 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.567656 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:31.567826 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.567711 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:39.567693852 +0000 UTC m=+17.221261436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:31.804041 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:31.803969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:31.804479 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.804109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:31.804479 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:31.804169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:31.804479 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:31.804354 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:32.805120 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:32.805025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:32.805543 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:32.805132 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:33.803871 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:33.803842 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:33.804049 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:33.803887 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:33.804049 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:33.803985 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:33.804176 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:33.804128 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:34.807050 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:34.807017 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:34.807464 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:34.807135 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:35.803247 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:35.803214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:35.803406 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:35.803257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:35.803406 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:35.803341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:35.803538 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:35.803499 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:36.803478 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:36.803441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:36.803966 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:36.803603 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:37.803604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:37.803573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:37.803999 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:37.803614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:37.803999 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:37.803697 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:37.803999 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:37.803829 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:38.803781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:38.803753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:38.804200 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:38.803843 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:39.536761 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:39.536727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:39.536988 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:39.536793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:39.536988 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.536893 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:39.536988 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.536981 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:55.536964199 +0000 UTC m=+33.190531775 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:39.537142 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.536893 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:39.537142 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.537079 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:55.53705626 +0000 UTC m=+33.190623850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:39.637315 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:39.637273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:39.637533 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.637439 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:39.637533 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.637474 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:39.637533 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.637485 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:39.637533 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.637533 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:55.637520526 +0000 UTC m=+33.291088103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:39.803788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:39.803703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:39.804259 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:39.803703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:39.804259 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.803839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:39.804259 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:39.803929 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:40.804259 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:40.804204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:40.804711 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:40.804339 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:41.804124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:41.804091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:41.804303 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:41.804214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:41.804303 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:41.804268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:41.804687 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:41.804384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:42.804382 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:42.804343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:42.804825 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:42.804463 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:43.803886 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:43.803462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:43.803886 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:43.803782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:43.804032 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:43.804012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:43.804087 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:43.804014 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:44.028500 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.028387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" event={"ID":"aa627ad35b77ae9bde45172877976b6e","Type":"ContainerStarted","Data":"a50d8fe0b1d1d9cbba91df6da8e636ac542f1e75b45e1f01dd333137732e0605"} Jul 06 19:27:44.033179 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.033153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"f400bfe953be7915875a895652a051b1ecb665e0b649267704d12a4ec5267549"} Jul 06 19:27:44.033266 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.033245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"bfab1ddee08a01838cb815174d8aaf202043a92c2f072e0b5587ace05f943c5e"} Jul 06 19:27:44.033304 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.033283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"bae1291cc3192b3a29d87c3953b247e33e9c5da6d265672af36a708641018321"} Jul 06 19:27:44.033359 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.033326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"b8c0dbcd96acd2fc3e2e163a48480ef3c062353c2521c57804d00d6f483cf77b"} Jul 06 19:27:44.034467 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.034447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" event={"ID":"af96c196-be64-4514-826b-4db41c7f1eca","Type":"ContainerStarted","Data":"c139bc69932eaf55e8b7f77e267bb95e1a39111a1c4224dcad8d2aa5d2f119b5"} Jul 06 19:27:44.035598 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.035581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pxqm" event={"ID":"85fbf08f-f42b-4610-bbf4-2808231b4550","Type":"ContainerStarted","Data":"5948806305bef3a2a3a945bbcbe46b76da590de20fb88f7401e70de67dc0e407"} Jul 06 19:27:44.116531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.116481 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-143.ec2.internal" podStartSLOduration=21.116465919 podStartE2EDuration="21.116465919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:44.078533533 +0000 UTC m=+21.732101131" watchObservedRunningTime="2026-07-06 19:27:44.116465919 +0000 UTC m=+21.770033517" Jul 06 19:27:44.157006 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.156963 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4pxqm" podStartSLOduration=2.830161087 podStartE2EDuration="22.156943235s" totalImagesPullingTime="19.326782148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:44.11633592 +0000 UTC m=+21.769903520" watchObservedRunningTime="2026-07-06 19:27:44.156943235 +0000 UTC m=+21.810510837" Jul 06 19:27:44.804090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:44.804059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:44.804224 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:44.804162 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:45.039602 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.039528 2577 generic.go:356] "Generic (PLEG): container finished" podID="944a10815a5caee6dab36f61ae5d0e90" containerID="2755c2165d9c70804bf8f59908dd16c844c51ba7534977a5f536fedf7bd9ac68" exitCode=0 Jul 06 19:27:45.040359 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.039600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" event={"ID":"944a10815a5caee6dab36f61ae5d0e90","Type":"ContainerDied","Data":"2755c2165d9c70804bf8f59908dd16c844c51ba7534977a5f536fedf7bd9ac68"} Jul 06 19:27:45.041061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.041035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7z4x5" event={"ID":"7ca5c635-0f15-4354-a416-513cbc89b7bb","Type":"ContainerStarted","Data":"5a006b63c27f1dae9e396a403166f265debdab1526179cec4332ce05cd4c56d9"} Jul 06 19:27:45.042478 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.042458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7qgxs" event={"ID":"ddf894a6-f277-4b6b-8306-ab44bdce81bf","Type":"ContainerStarted","Data":"dfe99b625e2fed3a121da8ee80bf6b8ea6fa5bddf13f3cb93bf4a32b4a87b062"} Jul 06 19:27:45.045264 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.045246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"8cb880338b05a822466f511c222c80e0bc5be3ac9c1b54e1f9da419e33e99791"} Jul 06 19:27:45.045337 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.045295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"96f00cffe3d0fa3a74447197a449fbe9de7afe1f513691c7ac512c6c3f303903"} Jul 06 19:27:45.046463 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.046445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xkrj6" event={"ID":"a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a","Type":"ContainerStarted","Data":"e241c83f1ca762b91198b3c00c0d522d155a8e6ff9ef787896b1aff7ab95433d"} Jul 06 19:27:45.047648 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.047630 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="5b957cdfd0872e24cb9023eeabf4d1dafd6588bc26189b0d0f73df439e3811e2" exitCode=0 Jul 06 19:27:45.047747 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.047689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"5b957cdfd0872e24cb9023eeabf4d1dafd6588bc26189b0d0f73df439e3811e2"} Jul 06 19:27:45.049100 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.049080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnw2t" event={"ID":"ccfd5d69-d911-484e-8b09-bcd602660c1a","Type":"ContainerStarted","Data":"7313a6b8f51223e0b33916930ac0a2bd830c68e18d924eca1e28caaf67749ab5"} Jul 06 19:27:45.051645 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.051264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" event={"ID":"327f5a30-a616-42ab-a70c-0bb02cfdd2d2","Type":"ContainerStarted","Data":"20b0f16bf45baaafd37dcdc109c227026841f445ea010f51f80eaac4e8b63d8e"} Jul 06 19:27:45.084304 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.084260 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2kvsm" podStartSLOduration=3.802408684 podStartE2EDuration="23.084245831s" totalImagesPullingTime="19.281837147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:44.157069686 +0000 UTC m=+21.810637285" watchObservedRunningTime="2026-07-06 19:27:45.084245831 +0000 UTC m=+22.737813475" Jul 06 19:27:45.115779 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.115303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7qgxs" podStartSLOduration=3.8652116100000002 podStartE2EDuration="23.115274781s" totalImagesPullingTime="19.250063171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:45.11418642 +0000 UTC m=+22.767754042" watchObservedRunningTime="2026-07-06 19:27:45.115274781 +0000 UTC m=+22.768842381" Jul 06 19:27:45.143196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.143137 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7z4x5" podStartSLOduration=3.897175556 podStartE2EDuration="23.143117857s" totalImagesPullingTime="19.245942301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:45.14305729 +0000 UTC m=+22.796624891" watchObservedRunningTime="2026-07-06 19:27:45.143117857 +0000 UTC m=+22.796685449" Jul 06 19:27:45.274780 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.274736 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnw2t" podStartSLOduration=4.006067224 podStartE2EDuration="23.274720951s" totalImagesPullingTime="19.268653727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:45.274688105 +0000 UTC m=+22.928255705" watchObservedRunningTime="2026-07-06 19:27:45.274720951 +0000 UTC m=+22.928288550" Jul 06 19:27:45.275181 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.275150 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xkrj6" podStartSLOduration=3.999266732 podStartE2EDuration="23.275140186s" totalImagesPullingTime="19.275873454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:45.227484774 +0000 UTC m=+22.881052374" watchObservedRunningTime="2026-07-06 19:27:45.275140186 +0000 UTC m=+22.928707784" Jul 06 19:27:45.617334 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.617313 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:27:45.755782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.755689 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:27:45.617329008Z","UUID":"9ffb2e41-329d-4fc0-9c64-ac68f3c8c65d","Handler":null,"Name":"","Endpoint":""} Jul 06 19:27:45.757776 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.757755 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:27:45.757931 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.757796 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:27:45.767408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.767377 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:45.767966 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.767859 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:45.803973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.803945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:45.804086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:45.803949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:45.804086 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:45.804047 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:45.804161 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:45.804116 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:46.055605 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.055526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" event={"ID":"327f5a30-a616-42ab-a70c-0bb02cfdd2d2","Type":"ContainerStarted","Data":"db1f263d40df18897003b405f4547dd006cf1de46f66954784bd87255160e2ac"} Jul 06 19:27:46.057805 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.057767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" event={"ID":"944a10815a5caee6dab36f61ae5d0e90","Type":"ContainerStarted","Data":"5127246922a1dc8bff8a4801d9d48a4fb827ac5e378481e61d98c5cc79700371"} Jul 06 19:27:46.058684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.058578 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:46.059169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.059152 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7qgxs" Jul 06 19:27:46.121172 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.121119 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-143.ec2.internal" podStartSLOduration=23.121098284 podStartE2EDuration="23.121098284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:46.092209141 +0000 UTC m=+23.745776741" watchObservedRunningTime="2026-07-06 19:27:46.121098284 +0000 UTC m=+23.774665885" Jul 06 19:27:46.804321 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:46.804240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:46.804481 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:46.804345 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:47.063366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:47.063277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"a18230c67f1b0eeb2a86c676403060b51e8cd2f5b2084d97878d6e81105b1eca"} Jul 06 19:27:47.065116 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:47.065081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" event={"ID":"327f5a30-a616-42ab-a70c-0bb02cfdd2d2","Type":"ContainerStarted","Data":"9ab63f6430e60e083efd01fb2595ed7b2a11bdf2cea165ae4e5f06ff31d68c50"} Jul 06 19:27:47.128347 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:47.128300 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dzcfd" podStartSLOduration=3.8122947959999998 podStartE2EDuration="25.128278392s" totalImagesPullingTime="21.315983596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:47.128226553 +0000 UTC m=+24.781794155" watchObservedRunningTime="2026-07-06 19:27:47.128278392 +0000 UTC m=+24.781845990" Jul 06 19:27:47.803474 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:47.803444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:47.803619 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:47.803454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:47.803619 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:47.803539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:47.803688 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:47.803669 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:48.803315 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:48.803292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:48.803720 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:48.803420 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:49.072956 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.072724 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" event={"ID":"e5d39682-b7a1-4d77-b913-3475037e2d43","Type":"ContainerStarted","Data":"220cd7d4965bb5fa5397cb2f5421d56e7e0d91875df18aa90d15e567f5e77fe1"} Jul 06 19:27:49.072956 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.072954 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:49.073155 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.072985 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:49.088095 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.088073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:49.188178 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.188135 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" podStartSLOduration=7.619530569 podStartE2EDuration="27.188121629s" totalImagesPullingTime="19.56859106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:49.144703933 +0000 UTC m=+26.798271532" watchObservedRunningTime="2026-07-06 19:27:49.188121629 +0000 UTC m=+26.841689228" Jul 06 19:27:49.804147 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.804110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:49.804636 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:49.804128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:49.804636 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:49.804245 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:49.804636 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:49.804278 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:50.078942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:50.078834 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:50.094049 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:50.094022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:27:50.804258 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:50.804231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:50.804788 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:50.804359 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:51.308859 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.308827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwq6w"] Jul 06 19:27:51.309045 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.308944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:51.309045 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:51.309029 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:51.341995 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.341968 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65kqw"] Jul 06 19:27:51.342111 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.342058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:51.342182 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:51.342161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:51.362792 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.362767 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jqwdl"] Jul 06 19:27:51.362881 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:51.362833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:51.362938 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:51.362916 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:52.085148 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:52.084834 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="59359b9ced8dde68b955a26dae947f090430695d7bb0b4985cf2afdb3c092eb1" exitCode=0 Jul 06 19:27:52.085858 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:52.084928 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"59359b9ced8dde68b955a26dae947f090430695d7bb0b4985cf2afdb3c092eb1"} Jul 06 19:27:52.804703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:52.804671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:52.804841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:52.804776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:52.804841 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:52.804792 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:52.804841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:52.804829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:52.804994 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:52.804913 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:52.805050 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:52.805009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:54.090823 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:54.090789 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="d5980e6aa2bdf1c3fd20dd77c1c068aeedc95bbec1433e8b4a43a299bc301fb5" exitCode=0 Jul 06 19:27:54.091213 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:54.090828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"d5980e6aa2bdf1c3fd20dd77c1c068aeedc95bbec1433e8b4a43a299bc301fb5"} Jul 06 19:27:54.803616 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:54.803585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:54.803821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:54.803585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:54.803821 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:54.803717 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:54.803821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:54.803585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:54.803821 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:54.803809 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:54.804047 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:54.803872 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:55.563050 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.562966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:55.563050 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.563029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:55.563417 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.563127 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:55.563417 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.563194 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs podName:4661f471-2b64-4e51-9361-f90b1dcc9910 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:27.563178777 +0000 UTC m=+65.216746355 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs") pod "network-metrics-daemon-xwq6w" (UID: "4661f471-2b64-4e51-9361-f90b1dcc9910") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:27:55.563417 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.563127 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:55.563417 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.563261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret podName:d11926aa-a0cc-40d1-9c68-17a44704d1b6 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:27.563249254 +0000 UTC m=+65.216816853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret") pod "global-pull-secret-syncer-65kqw" (UID: "d11926aa-a0cc-40d1-9c68-17a44704d1b6") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:27:55.654998 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.654959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:55.664019 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.663996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:55.664152 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.664140 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:27:55.664186 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.664158 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:27:55.664186 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.664167 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lp7v8 for pod openshift-network-diagnostics/network-check-target-jqwdl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:55.664244 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:55.664209 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8 podName:69d348dd-bd01-4286-98a6-3d5ba397a825 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:27.664198124 +0000 UTC m=+65.317765705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lp7v8" (UniqueName: "kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8") pod "network-check-target-jqwdl" (UID: "69d348dd-bd01-4286-98a6-3d5ba397a825") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:27:55.764841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.764810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:55.832767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:55.832695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:56.097645 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:56.097554 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="ae9620e38e5d75358e3191069a3d83af3164e54a09f90f910ba2c218923ee3ba" exitCode=0 Jul 06 19:27:56.097645 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:56.097595 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"ae9620e38e5d75358e3191069a3d83af3164e54a09f90f910ba2c218923ee3ba"} Jul 06 19:27:56.804052 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:56.804019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:56.804052 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:56.804046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:56.804645 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:56.804020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:56.804645 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:56.804161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwq6w" podUID="4661f471-2b64-4e51-9361-f90b1dcc9910" Jul 06 19:27:56.804645 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:56.804262 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jqwdl" podUID="69d348dd-bd01-4286-98a6-3d5ba397a825" Jul 06 19:27:56.804645 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:56.804343 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65kqw" podUID="d11926aa-a0cc-40d1-9c68-17a44704d1b6" Jul 06 19:27:57.129056 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.129017 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-143.ec2.internal" event="NodeReady" Jul 06 19:27:57.129231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.129175 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:27:57.167922 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.167873 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84d69ccdc4-tv2kz"] Jul 06 19:27:57.183711 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.183684 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7nxzj"] Jul 06 19:27:57.183849 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.183788 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.187138 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.187118 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wrxff\"" Jul 06 19:27:57.187138 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.187119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:27:57.187399 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.187377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:27:57.187596 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.187580 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:27:57.193999 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.193977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84d69ccdc4-tv2kz"] Jul 06 19:27:57.194107 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.194014 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5dgxw"] Jul 06 19:27:57.194107 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.194096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.194213 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.194189 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:27:57.197425 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.197406 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6kw96\"" Jul 06 19:27:57.197667 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.197652 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:27:57.197717 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.197689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:27:57.212021 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.211999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dgxw"] Jul 06 19:27:57.212129 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.212033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nxzj"] Jul 06 19:27:57.212166 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.212143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.215358 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.215342 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:27:57.215450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.215344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qg2nr\"" Jul 06 19:27:57.216043 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.216027 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:27:57.216375 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.216362 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:27:57.274788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.274788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.274788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.275057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.275057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.275057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.274960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.275057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.275007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5h24\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.275057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.275033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376067 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.375989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5h24\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376067 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5d4t\" (UniqueName: \"kubernetes.io/projected/8f6910e1-7652-4261-a307-3fbdc052ff9e-kube-api-access-z5d4t\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fadcafe9-3b20-40c4-b1a8-15860c940eec-config-volume\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.376287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngmc\" (UniqueName: \"kubernetes.io/projected/fadcafe9-3b20-40c4-b1a8-15860c940eec-kube-api-access-rngmc\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.376357 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.376375 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.376436 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:27:57.87641442 +0000 UTC m=+35.529982010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6910e1-7652-4261-a307-3fbdc052ff9e-cert\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadcafe9-3b20-40c4-b1a8-15860c940eec-tmp-dir\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.376770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.376764 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.377154 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.377135 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.380528 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.380429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.380584 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.380485 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.390981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.390955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.391109 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.391010 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5h24\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.477957 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.477918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fadcafe9-3b20-40c4-b1a8-15860c940eec-config-volume\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.477957 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.477959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngmc\" (UniqueName: \"kubernetes.io/projected/fadcafe9-3b20-40c4-b1a8-15860c940eec-kube-api-access-rngmc\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.478190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.477998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6910e1-7652-4261-a307-3fbdc052ff9e-cert\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.478190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.478018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadcafe9-3b20-40c4-b1a8-15860c940eec-tmp-dir\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.478190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.478064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.478190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.478101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5d4t\" (UniqueName: \"kubernetes.io/projected/8f6910e1-7652-4261-a307-3fbdc052ff9e-kube-api-access-z5d4t\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.478389 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.478194 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:57.478389 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.478300 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls podName:fadcafe9-3b20-40c4-b1a8-15860c940eec nodeName:}" failed. No retries permitted until 2026-07-06 19:27:57.978277781 +0000 UTC m=+35.631845368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls") pod "dns-default-7nxzj" (UID: "fadcafe9-3b20-40c4-b1a8-15860c940eec") : secret "dns-default-metrics-tls" not found Jul 06 19:27:57.478497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.478385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fadcafe9-3b20-40c4-b1a8-15860c940eec-tmp-dir\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.480582 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.480557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6910e1-7652-4261-a307-3fbdc052ff9e-cert\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.488482 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.488457 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fadcafe9-3b20-40c4-b1a8-15860c940eec-config-volume\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.489327 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.489306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngmc\" (UniqueName: \"kubernetes.io/projected/fadcafe9-3b20-40c4-b1a8-15860c940eec-kube-api-access-rngmc\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.489525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.489510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5d4t\" (UniqueName: \"kubernetes.io/projected/8f6910e1-7652-4261-a307-3fbdc052ff9e-kube-api-access-z5d4t\") pod \"ingress-canary-5dgxw\" (UID: \"8f6910e1-7652-4261-a307-3fbdc052ff9e\") " pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.520159 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.520133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dgxw" Jul 06 19:27:57.705312 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.705281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dgxw"] Jul 06 19:27:57.708872 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:27:57.708839 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6910e1_7652_4261_a307_3fbdc052ff9e.slice/crio-92875cdbdc3dbfa22330174ffe90e60c56a894e68a6df94efc1abcfd0438fef4 WatchSource:0}: Error finding container 92875cdbdc3dbfa22330174ffe90e60c56a894e68a6df94efc1abcfd0438fef4: Status 404 returned error can't find the container with id 92875cdbdc3dbfa22330174ffe90e60c56a894e68a6df94efc1abcfd0438fef4 Jul 06 19:27:57.881431 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.881397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:57.882027 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.881577 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:27:57.882027 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.881601 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:27:57.882027 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.881671 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:27:58.881650919 +0000 UTC m=+36.535218507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:27:57.982728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:57.982639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:57.982880 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.982791 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:57.982952 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:57.982923 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls podName:fadcafe9-3b20-40c4-b1a8-15860c940eec nodeName:}" failed. No retries permitted until 2026-07-06 19:27:58.982883291 +0000 UTC m=+36.636450875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls") pod "dns-default-7nxzj" (UID: "fadcafe9-3b20-40c4-b1a8-15860c940eec") : secret "dns-default-metrics-tls" not found Jul 06 19:27:58.103195 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.103161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dgxw" event={"ID":"8f6910e1-7652-4261-a307-3fbdc052ff9e","Type":"ContainerStarted","Data":"92875cdbdc3dbfa22330174ffe90e60c56a894e68a6df94efc1abcfd0438fef4"} Jul 06 19:27:58.771129 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.771091 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl"] Jul 06 19:27:58.792443 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.792249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl"] Jul 06 19:27:58.792443 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.792358 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:58.795503 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.795477 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bksgq\"" Jul 06 19:27:58.795744 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.795728 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:27:58.796005 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.795813 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:27:58.804223 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.804203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:27:58.804321 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.804204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:27:58.804406 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.804377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:27:58.807711 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.807365 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:27:58.807815 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.807791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:27:58.808485 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.808465 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdwjn\"" Jul 06 19:27:58.808706 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.808691 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjmjf\"" Jul 06 19:27:58.808780 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.808728 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:27:58.808963 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.808948 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:27:58.891508 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.891478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.891542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de34a387-7a02-4be4-8c42-aba4d0989fb1-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.891583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.891658 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.891676 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.891730 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:28:00.891711777 +0000 UTC m=+38.545279369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:27:58.892117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.891783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:58.993004 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.992970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:27:58.993188 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.993056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de34a387-7a02-4be4-8c42-aba4d0989fb1-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:58.993188 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.993091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:58.993188 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.993119 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:58.993351 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.993193 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls podName:fadcafe9-3b20-40c4-b1a8-15860c940eec nodeName:}" failed. No retries permitted until 2026-07-06 19:28:00.993170452 +0000 UTC m=+38.646738049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls") pod "dns-default-7nxzj" (UID: "fadcafe9-3b20-40c4-b1a8-15860c940eec") : secret "dns-default-metrics-tls" not found Jul 06 19:27:58.993453 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.993422 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:27:58.993604 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:58.993492 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:59.493472736 +0000 UTC m=+37.147040333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:27:58.993997 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:58.993975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de34a387-7a02-4be4-8c42-aba4d0989fb1-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:59.497762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:27:59.497725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:27:59.497943 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:59.497870 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:27:59.498014 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:27:59.497958 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:00.497938727 +0000 UTC m=+38.151506306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:28:00.506473 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:00.506440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:00.507002 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:00.506573 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:28:00.507002 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:00.506638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:02.506621255 +0000 UTC m=+40.160188832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:28:00.682037 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:00.681994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:00.781594 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:00.781517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:00.909180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:00.909141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:00.909375 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:00.909288 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:28:00.909375 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:00.909304 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:28:00.909479 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:00.909378 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:28:04.909355015 +0000 UTC m=+42.562922607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:28:00.975864 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:00.975809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:01.009965 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:01.009927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:01.010151 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:01.010064 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:28:01.010151 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:01.010137 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls podName:fadcafe9-3b20-40c4-b1a8-15860c940eec nodeName:}" failed. No retries permitted until 2026-07-06 19:28:05.010118019 +0000 UTC m=+42.663685601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls") pod "dns-default-7nxzj" (UID: "fadcafe9-3b20-40c4-b1a8-15860c940eec") : secret "dns-default-metrics-tls" not found Jul 06 19:28:01.076952 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:01.076831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:01.860516 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:01.860491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.107048 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.106954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.115958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.115932 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="d8dcbac619dc614d79836651aa0ca205eec4076962c58ae5821fa66719955f67" exitCode=0 Jul 06 19:28:02.116067 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.116014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"d8dcbac619dc614d79836651aa0ca205eec4076962c58ae5821fa66719955f67"} Jul 06 19:28:02.117522 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.117499 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dgxw" event={"ID":"8f6910e1-7652-4261-a307-3fbdc052ff9e","Type":"ContainerStarted","Data":"0c5a9a53d69deb79589dcc828ac78d9f4f97a38d117543757b648ec8572f1424"} Jul 06 19:28:02.118858 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.118838 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-clz5w"] Jul 06 19:28:02.139201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.139178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.142589 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.142567 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmc6g\"" Jul 06 19:28:02.142751 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.142731 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:28:02.142846 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.142754 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:28:02.142846 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.142788 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:28:02.143064 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.143050 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:28:02.150355 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.150329 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clz5w"] Jul 06 19:28:02.211583 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.211549 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5dgxw" podStartSLOduration=1.08747701 podStartE2EDuration="5.211536299s" totalImagesPullingTime="4.124059289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:02.210597852 +0000 UTC m=+39.864165451" watchObservedRunningTime="2026-07-06 19:28:02.211536299 +0000 UTC m=+39.865103898" Jul 06 19:28:02.222878 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.222855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.291248 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.291227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.319006 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.318984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db58b7de-f080-402f-9ad4-e50c230c3e8f-crio-socket\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.319078 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.319015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db58b7de-f080-402f-9ad4-e50c230c3e8f-data-volume\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.319078 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.319058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.319150 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.319110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckc2r\" (UniqueName: \"kubernetes.io/projected/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-api-access-ckc2r\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.319183 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.319156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419534 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckc2r\" (UniqueName: \"kubernetes.io/projected/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-api-access-ckc2r\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419534 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db58b7de-f080-402f-9ad4-e50c230c3e8f-crio-socket\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db58b7de-f080-402f-9ad4-e50c230c3e8f-data-volume\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419754 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.419706 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:02.419802 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.419758 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:02.919744308 +0000 UTC m=+40.573311899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:02.419802 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419773 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/db58b7de-f080-402f-9ad4-e50c230c3e8f-crio-socket\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419932 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.419814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.419932 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.419923 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:02.420031 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.419970 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:02.919958809 +0000 UTC m=+40.573526386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:02.420085 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.420051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/db58b7de-f080-402f-9ad4-e50c230c3e8f-data-volume\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.429667 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.429651 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckc2r\" (UniqueName: \"kubernetes.io/projected/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-api-access-ckc2r\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.520563 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.520539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:02.520650 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.520628 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:28:02.520696 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.520669 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:06.52065898 +0000 UTC m=+44.174226557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:28:02.613885 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.613861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.731196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.731175 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d"] Jul 06 19:28:02.749883 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.749866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d"] Jul 06 19:28:02.749992 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.749966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" Jul 06 19:28:02.753130 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.753109 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tn2zf\"" Jul 06 19:28:02.753228 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.753111 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 19:28:02.753228 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.753112 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 19:28:02.756971 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.756938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.850304 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.850280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:02.923684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.923657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.923707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69b4b\" (UniqueName: \"kubernetes.io/projected/c7f75bce-0a16-4a12-b137-cbb82dd0b5d7-kube-api-access-69b4b\") pod \"migrator-d57d58cc9-lk95d\" (UID: \"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.923767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.923810 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.923855 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.923861 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:03.92384788 +0000 UTC m=+41.577415457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:02.924048 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:02.923929 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:03.923910482 +0000 UTC m=+41.577478062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:02.948694 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:02.948672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:03.024945 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.024881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69b4b\" (UniqueName: \"kubernetes.io/projected/c7f75bce-0a16-4a12-b137-cbb82dd0b5d7-kube-api-access-69b4b\") pod \"migrator-d57d58cc9-lk95d\" (UID: \"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" Jul 06 19:28:03.034311 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.034287 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69b4b\" (UniqueName: \"kubernetes.io/projected/c7f75bce-0a16-4a12-b137-cbb82dd0b5d7-kube-api-access-69b4b\") pod \"migrator-d57d58cc9-lk95d\" (UID: \"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" Jul 06 19:28:03.059249 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.059229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" Jul 06 19:28:03.129020 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.128985 2577 generic.go:356] "Generic (PLEG): container finished" podID="a8830363-3b24-4e4b-94e9-635775bc5552" containerID="7e29b1710c38380bce07fe5e55ea0e05a3cf183b955ee7bbfc2a0242eb6ea8c8" exitCode=0 Jul 06 19:28:03.129126 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.129109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerDied","Data":"7e29b1710c38380bce07fe5e55ea0e05a3cf183b955ee7bbfc2a0242eb6ea8c8"} Jul 06 19:28:03.212546 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.212513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d"] Jul 06 19:28:03.215256 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:03.215235 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f75bce_0a16_4a12_b137_cbb82dd0b5d7.slice/crio-75df15a671f61faa42255651382bc413cafa46e716252c2162217e167dea43bb WatchSource:0}: Error finding container 75df15a671f61faa42255651382bc413cafa46e716252c2162217e167dea43bb: Status 404 returned error can't find the container with id 75df15a671f61faa42255651382bc413cafa46e716252c2162217e167dea43bb Jul 06 19:28:03.807915 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.807848 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xkrj6_a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a/dns-node-resolver/0.log" Jul 06 19:28:03.931836 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.931797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:03.932289 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:03.931917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:03.932289 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:03.931976 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:03.932289 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:03.932045 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:03.932289 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:03.932071 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:05.932049618 +0000 UTC m=+43.585617218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:03.932289 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:03.932113 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:05.932095411 +0000 UTC m=+43.585662994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:04.134564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.134487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" event={"ID":"a8830363-3b24-4e4b-94e9-635775bc5552","Type":"ContainerStarted","Data":"0e3e8a2f94a9f4d15d22bbb633e57dcf88c157911e42c05e8aab81ed7c207a71"} Jul 06 19:28:04.135533 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.135510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" event={"ID":"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7","Type":"ContainerStarted","Data":"75df15a671f61faa42255651382bc413cafa46e716252c2162217e167dea43bb"} Jul 06 19:28:04.151612 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.151589 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-492x6"] Jul 06 19:28:04.164012 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.163834 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8cn5l" podStartSLOduration=8.160530194 podStartE2EDuration="42.163815985s" totalImagesPullingTime="34.003285791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:04.162256472 +0000 UTC m=+41.815824097" watchObservedRunningTime="2026-07-06 19:28:04.163815985 +0000 UTC m=+41.817383583" Jul 06 19:28:04.177891 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.177869 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-492x6"] Jul 06 19:28:04.178001 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.177977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.181258 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.181235 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 19:28:04.181681 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.181661 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 19:28:04.181777 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.181750 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 19:28:04.181777 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.181761 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ft9tn\"" Jul 06 19:28:04.181885 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.181782 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 19:28:04.199463 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.199435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:04.233582 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.233557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-key\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.233678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.233628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-cabundle\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.233678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.233659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qp59\" (UniqueName: \"kubernetes.io/projected/75676fd4-64b1-4228-aa1a-8283c98ea0bd-kube-api-access-6qp59\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.287726 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.287701 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:04.334243 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.334223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-cabundle\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.334366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.334250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qp59\" (UniqueName: \"kubernetes.io/projected/75676fd4-64b1-4228-aa1a-8283c98ea0bd-kube-api-access-6qp59\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.334487 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.334463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-key\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.335062 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.335035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-cabundle\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.337270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.337236 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75676fd4-64b1-4228-aa1a-8283c98ea0bd-signing-key\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.344485 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.344466 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qp59\" (UniqueName: \"kubernetes.io/projected/75676fd4-64b1-4228-aa1a-8283c98ea0bd-kube-api-access-6qp59\") pod \"service-ca-758f599c5-492x6\" (UID: \"75676fd4-64b1-4228-aa1a-8283c98ea0bd\") " pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.389014 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.388957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:04.488825 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.488803 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-492x6" Jul 06 19:28:04.607661 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.607640 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7z4x5_7ca5c635-0f15-4354-a416-513cbc89b7bb/node-ca/0.log" Jul 06 19:28:04.648253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.648186 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-492x6"] Jul 06 19:28:04.824197 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:04.824170 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75676fd4_64b1_4228_aa1a_8283c98ea0bd.slice/crio-c717dc2e7230362fdd6459df68f4ef045e53096c1c09e669d27640baa078f104 WatchSource:0}: Error finding container c717dc2e7230362fdd6459df68f4ef045e53096c1c09e669d27640baa078f104: Status 404 returned error can't find the container with id c717dc2e7230362fdd6459df68f4ef045e53096c1c09e669d27640baa078f104 Jul 06 19:28:04.938571 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:04.938548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:04.938861 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:04.938685 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:28:04.938861 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:04.938701 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:28:04.938861 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:04.938750 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:28:12.938735893 +0000 UTC m=+50.592303470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:28:05.038859 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.038837 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:05.039027 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.039010 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:28:05.039080 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.039072 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls podName:fadcafe9-3b20-40c4-b1a8-15860c940eec nodeName:}" failed. No retries permitted until 2026-07-06 19:28:13.039058285 +0000 UTC m=+50.692625862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls") pod "dns-default-7nxzj" (UID: "fadcafe9-3b20-40c4-b1a8-15860c940eec") : secret "dns-default-metrics-tls" not found Jul 06 19:28:05.140135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.140097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" event={"ID":"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7","Type":"ContainerStarted","Data":"153be5e92694e7a4ee68ab44867a19d5a824cc59f719c6d8760ddd77578a9c24"} Jul 06 19:28:05.140289 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.140179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" event={"ID":"c7f75bce-0a16-4a12-b137-cbb82dd0b5d7","Type":"ContainerStarted","Data":"f02b23d98d0d72fe1445e48dc917e67b0beb2d2f447f3d1b66bd242a4911d102"} Jul 06 19:28:05.141303 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.141279 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-492x6" event={"ID":"75676fd4-64b1-4228-aa1a-8283c98ea0bd","Type":"ContainerStarted","Data":"c717dc2e7230362fdd6459df68f4ef045e53096c1c09e669d27640baa078f104"} Jul 06 19:28:05.188139 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.188060 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-lk95d" podStartSLOduration=1.5465124989999999 podStartE2EDuration="3.188045287s" totalImagesPullingTime="1.641532788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:05.187267734 +0000 UTC m=+42.840835356" watchObservedRunningTime="2026-07-06 19:28:05.188045287 +0000 UTC m=+42.841612877" Jul 06 19:28:05.215521 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.215503 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5dgxw_8f6910e1-7652-4261-a307-3fbdc052ff9e/serve-healthcheck-canary/0.log" Jul 06 19:28:05.317465 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.317430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:05.944720 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.944684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:05.945170 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:05.944767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:05.945170 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.944840 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:05.945170 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.944890 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:05.945170 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.944958 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:09.944938445 +0000 UTC m=+47.598506036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:05.945170 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:05.944978 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:09.944969279 +0000 UTC m=+47.598536884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:06.549117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:06.549086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:06.549270 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:06.549204 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:28:06.549270 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:06.549269 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:14.549256612 +0000 UTC m=+52.202824188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:28:09.151564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.151525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-492x6" event={"ID":"75676fd4-64b1-4228-aa1a-8283c98ea0bd","Type":"ContainerStarted","Data":"fa3d585c5806f5718d449e5e88d061a40e5d6864828e91221bb5d7566e718fe1"} Jul 06 19:28:09.185283 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.185239 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-492x6" podStartSLOduration=1.909602408 podStartE2EDuration="5.185219893s" totalImagesPullingTime="3.275617485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:09.185136212 +0000 UTC m=+46.838703807" watchObservedRunningTime="2026-07-06 19:28:09.185219893 +0000 UTC m=+46.838787492" Jul 06 19:28:09.315492 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.315464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.524381 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.524320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.633496 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.633464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.725887 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.725849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.832857 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.832782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.930559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.930531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:09.976849 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.976822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:09.976979 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:09.976883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:09.976979 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:09.976955 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:09.977054 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:09.976983 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:09.977054 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:09.977000 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:17.976987213 +0000 UTC m=+55.630554791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:09.977054 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:09.977037 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:17.977018435 +0000 UTC m=+55.630586013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:10.854595 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:10.854560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:11.009472 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:11.009434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:11.203035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:11.202946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:11.608251 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:11.608219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:12.000587 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:12.000557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:12.214045 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:12.214016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:12.814595 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:12.814564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:13.003451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.003414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") pod \"image-registry-84d69ccdc4-tv2kz\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:13.004140 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:13.003595 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:28:13.004140 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:13.003613 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84d69ccdc4-tv2kz: secret "image-registry-tls" not found Jul 06 19:28:13.004140 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:13.003682 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls podName:919dec9b-48ae-419f-b437-0627ccfb100a nodeName:}" failed. No retries permitted until 2026-07-06 19:28:29.003661593 +0000 UTC m=+66.657229173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls") pod "image-registry-84d69ccdc4-tv2kz" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a") : secret "image-registry-tls" not found Jul 06 19:28:13.038637 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.038610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:13.104066 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.103988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:13.106335 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.106307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fadcafe9-3b20-40c4-b1a8-15860c940eec-metrics-tls\") pod \"dns-default-7nxzj\" (UID: \"fadcafe9-3b20-40c4-b1a8-15860c940eec\") " pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:13.229983 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.229949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:13.402949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.402863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:13.452798 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.452751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:13.527383 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.527355 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nxzj"] Jul 06 19:28:13.530059 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:13.530037 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadcafe9_3b20_40c4_b1a8_15860c940eec.slice/crio-02345c24fd980a423f2ac6fd32993ab1f9f015eee688166c7d36e7e63b5dc7aa WatchSource:0}: Error finding container 02345c24fd980a423f2ac6fd32993ab1f9f015eee688166c7d36e7e63b5dc7aa: Status 404 returned error can't find the container with id 02345c24fd980a423f2ac6fd32993ab1f9f015eee688166c7d36e7e63b5dc7aa Jul 06 19:28:13.532039 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.532015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:13.608577 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:13.608538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:14.016160 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:14.016127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:14.127618 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:14.127584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:14.165803 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:14.165769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nxzj" event={"ID":"fadcafe9-3b20-40c4-b1a8-15860c940eec","Type":"ContainerStarted","Data":"02345c24fd980a423f2ac6fd32993ab1f9f015eee688166c7d36e7e63b5dc7aa"} Jul 06 19:28:14.606862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:14.606830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:14.617176 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:14.617147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:14.617346 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:14.617324 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:28:14.617476 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:14.617410 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert podName:de34a387-7a02-4be4-8c42-aba4d0989fb1 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:30.617390594 +0000 UTC m=+68.270958171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-l7gvl" (UID: "de34a387-7a02-4be4-8c42-aba4d0989fb1") : secret "networking-console-plugin-cert" not found Jul 06 19:28:15.017047 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:15.017009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:15.169112 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:15.169079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:15.274660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:15.274586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:15.837234 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:15.837156 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:15.940355 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:15.940323 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:16.173652 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.173563 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nxzj" event={"ID":"fadcafe9-3b20-40c4-b1a8-15860c940eec","Type":"ContainerStarted","Data":"4b283764b875bfe379f3cfa3e42330ef86ac16f0f39ed3a32de3c1bdc46b3ca1"} Jul 06 19:28:16.173652 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.173615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nxzj" event={"ID":"fadcafe9-3b20-40c4-b1a8-15860c940eec","Type":"ContainerStarted","Data":"625c21c8ab042ec4edec0575d12011d519181dc9d0677c58b0fab538df3865a1"} Jul 06 19:28:16.174253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.173702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:16.198155 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.198097 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7nxzj" podStartSLOduration=17.208094556 podStartE2EDuration="19.198075667s" totalImagesPullingTime="1.989981111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:16.197925063 +0000 UTC m=+53.851492664" watchObservedRunningTime="2026-07-06 19:28:16.198075667 +0000 UTC m=+53.851643270" Jul 06 19:28:16.222045 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.222017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:16.455266 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:16.455190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.027489 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:17.027460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.268139 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:17.268110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.412169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:17.412086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.824111 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:17.824081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:18.044916 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:18.044868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:18.045079 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:18.044985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:18.045131 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:18.045091 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.045175 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:18.045163 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls podName:db58b7de-f080-402f-9ad4-e50c230c3e8f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:34.045142861 +0000 UTC m=+71.698710440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-clz5w" (UID: "db58b7de-f080-402f-9ad4-e50c230c3e8f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.045496 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:18.045476 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/db58b7de-f080-402f-9ad4-e50c230c3e8f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:18.241992 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:18.241959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:18.606499 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:18.606419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:19.049125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:19.049090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:19.221648 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:19.221617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:19.600912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:19.600873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:20.009431 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:20.009406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:20.207280 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:20.207256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:22.096257 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:22.096230 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbm82" Jul 06 19:28:25.609741 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:25.609705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:25.709271 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:25.709230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.112719 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.112686 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48"] Jul 06 19:28:26.116792 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.116341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf"] Jul 06 19:28:26.116959 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.116916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.118681 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.118658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.124214 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.124181 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:28:26.124290 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.124197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:28:26.124361 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.124201 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:28:26.125629 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.125613 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:28:26.125721 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.125708 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-d68tm\"" Jul 06 19:28:26.125776 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.125752 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:28:26.129559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.129537 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48"] Jul 06 19:28:26.137554 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.137527 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf"] Jul 06 19:28:26.156428 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.156397 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84d69ccdc4-tv2kz"] Jul 06 19:28:26.156612 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:26.156595 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" podUID="919dec9b-48ae-419f-b437-0627ccfb100a" Jul 06 19:28:26.180309 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.180277 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7nxzj" Jul 06 19:28:26.206296 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.206273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:26.208820 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.208794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgws\" (UniqueName: \"kubernetes.io/projected/af2b7195-c315-469b-b363-d44607a58d65-kube-api-access-jsgws\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.208955 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.208847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af2b7195-c315-469b-b363-d44607a58d65-klusterlet-config\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.208955 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.208931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44bh\" (UniqueName: \"kubernetes.io/projected/18ebce76-5361-4755-b835-30fe160b155e-kube-api-access-c44bh\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.209069 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.208980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af2b7195-c315-469b-b363-d44607a58d65-tmp\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.209069 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.209058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ebce76-5361-4755-b835-30fe160b155e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.211668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.211648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:26.219914 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.219878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf4c59977-qr592"] Jul 06 19:28:26.221646 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.221628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.239068 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.239038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.248274 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.248247 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf4c59977-qr592"] Jul 06 19:28:26.309462 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309432 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309462 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309466 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5h24\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309508 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309530 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309571 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309592 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration\") pod \"919dec9b-48ae-419f-b437-0627ccfb100a\" (UID: \"919dec9b-48ae-419f-b437-0627ccfb100a\") " Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgws\" (UniqueName: \"kubernetes.io/projected/af2b7195-c315-469b-b363-d44607a58d65-kube-api-access-jsgws\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b20d746-f135-428f-bee1-f662be0b9174-ca-trust-extracted\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af2b7195-c315-469b-b363-d44607a58d65-klusterlet-config\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-installation-pull-secrets\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44bh\" (UniqueName: \"kubernetes.io/projected/18ebce76-5361-4755-b835-30fe160b155e-kube-api-access-c44bh\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.309938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309929 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:28:26.310257 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af2b7195-c315-469b-b363-d44607a58d65-tmp\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.310257 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309972 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:26.310257 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.309992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-registry-tls\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-bound-sa-token\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-registry-certificates\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-image-registry-private-configuration\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ebce76-5361-4755-b835-30fe160b155e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.310515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkq7x\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-kube-api-access-zkq7x\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-trusted-ca\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.310655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310582 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/919dec9b-48ae-419f-b437-0627ccfb100a-ca-trust-extracted\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.310655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310586 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af2b7195-c315-469b-b363-d44607a58d65-tmp\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.310655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310599 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-trusted-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.310861 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.310687 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:28:26.312106 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.312078 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:28:26.312292 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.312272 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:28:26.312469 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.312449 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:28:26.312560 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.312541 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24" (OuterVolumeSpecName: "kube-api-access-h5h24") pod "919dec9b-48ae-419f-b437-0627ccfb100a" (UID: "919dec9b-48ae-419f-b437-0627ccfb100a"). InnerVolumeSpecName "kube-api-access-h5h24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:28:26.312867 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.312844 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af2b7195-c315-469b-b363-d44607a58d65-klusterlet-config\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.313205 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.313188 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ebce76-5361-4755-b835-30fe160b155e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.322722 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.322694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgws\" (UniqueName: \"kubernetes.io/projected/af2b7195-c315-469b-b363-d44607a58d65-kube-api-access-jsgws\") pod \"klusterlet-addon-workmgr-675c7796bf-gfq48\" (UID: \"af2b7195-c315-469b-b363-d44607a58d65\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.322834 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.322801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44bh\" (UniqueName: \"kubernetes.io/projected/18ebce76-5361-4755-b835-30fe160b155e-kube-api-access-c44bh\") pod \"managed-serviceaccount-addon-agent-964d89bdd-9zlvf\" (UID: \"18ebce76-5361-4755-b835-30fe160b155e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.344194 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.344149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.411942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.411833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-bound-sa-token\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.411942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.411868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-registry-certificates\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.411942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.411922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-image-registry-private-configuration\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412219 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkq7x\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-kube-api-access-zkq7x\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412219 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-trusted-ca\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412219 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b20d746-f135-428f-bee1-f662be0b9174-ca-trust-extracted\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412219 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-installation-pull-secrets\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-registry-tls\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412319 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-bound-sa-token\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412336 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/919dec9b-48ae-419f-b437-0627ccfb100a-registry-certificates\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412351 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-image-registry-private-configuration\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412369 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5h24\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-kube-api-access-h5h24\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.412415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412385 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/919dec9b-48ae-419f-b437-0627ccfb100a-installation-pull-secrets\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:26.412684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b20d746-f135-428f-bee1-f662be0b9174-ca-trust-extracted\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.412930 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.412888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-registry-certificates\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.413142 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.413123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b20d746-f135-428f-bee1-f662be0b9174-trusted-ca\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.414393 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.414369 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-image-registry-private-configuration\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.414547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.414531 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b20d746-f135-428f-bee1-f662be0b9174-installation-pull-secrets\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.414662 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.414647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-registry-tls\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.424551 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.424533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-bound-sa-token\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.426995 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.426976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:26.427285 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.427269 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkq7x\" (UniqueName: \"kubernetes.io/projected/9b20d746-f135-428f-bee1-f662be0b9174-kube-api-access-zkq7x\") pod \"image-registry-5cf4c59977-qr592\" (UID: \"9b20d746-f135-428f-bee1-f662be0b9174\") " pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.434785 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.434763 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.438926 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.438889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" Jul 06 19:28:26.535986 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.535964 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wrxff\"" Jul 06 19:28:26.536233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.536209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.541058 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.541030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:26.556279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.556255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48"] Jul 06 19:28:26.558912 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:26.558880 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2b7195_c315_469b_b363_d44607a58d65.slice/crio-ceb78763d851e92050a9ce8959f12ee97cb92f407a12126a03f104ea7f7de93d WatchSource:0}: Error finding container ceb78763d851e92050a9ce8959f12ee97cb92f407a12126a03f104ea7f7de93d: Status 404 returned error can't find the container with id ceb78763d851e92050a9ce8959f12ee97cb92f407a12126a03f104ea7f7de93d Jul 06 19:28:26.575183 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.575160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf"] Jul 06 19:28:26.578164 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:26.578144 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18ebce76_5361_4755_b835_30fe160b155e.slice/crio-513e245f6bd9ec6109aabf9041c4d45a808e14ad2946c36db06556669414d6cd WatchSource:0}: Error finding container 513e245f6bd9ec6109aabf9041c4d45a808e14ad2946c36db06556669414d6cd: Status 404 returned error can't find the container with id 513e245f6bd9ec6109aabf9041c4d45a808e14ad2946c36db06556669414d6cd Jul 06 19:28:26.669651 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:26.669624 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf4c59977-qr592"] Jul 06 19:28:26.671841 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:26.671809 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b20d746_f135_428f_bee1_f662be0b9174.slice/crio-c960dab80e2d2d68c9e7166d0ed3379f6a34747e4bb2da43f0940704654a4321 WatchSource:0}: Error finding container c960dab80e2d2d68c9e7166d0ed3379f6a34747e4bb2da43f0940704654a4321: Status 404 returned error can't find the container with id c960dab80e2d2d68c9e7166d0ed3379f6a34747e4bb2da43f0940704654a4321 Jul 06 19:28:27.102595 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.102564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.162280 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.162249 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.211050 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.211020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" event={"ID":"9b20d746-f135-428f-bee1-f662be0b9174","Type":"ContainerStarted","Data":"11b97a5b14bd087c9fa16bf60ae6e6d1eb5b759c9cd9d5f809e78ee9fd6707ab"} Jul 06 19:28:27.211218 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.211087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" event={"ID":"9b20d746-f135-428f-bee1-f662be0b9174","Type":"ContainerStarted","Data":"c960dab80e2d2d68c9e7166d0ed3379f6a34747e4bb2da43f0940704654a4321"} Jul 06 19:28:27.211218 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.211126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:27.212318 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.212294 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" event={"ID":"18ebce76-5361-4755-b835-30fe160b155e","Type":"ContainerStarted","Data":"513e245f6bd9ec6109aabf9041c4d45a808e14ad2946c36db06556669414d6cd"} Jul 06 19:28:27.213401 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.213376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" event={"ID":"af2b7195-c315-469b-b363-d44607a58d65","Type":"ContainerStarted","Data":"ceb78763d851e92050a9ce8959f12ee97cb92f407a12126a03f104ea7f7de93d"} Jul 06 19:28:27.213401 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.213393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84d69ccdc4-tv2kz" Jul 06 19:28:27.234487 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.234444 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" podStartSLOduration=1.234431154 podStartE2EDuration="1.234431154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:27.234219506 +0000 UTC m=+64.887787106" watchObservedRunningTime="2026-07-06 19:28:27.234431154 +0000 UTC m=+64.887998743" Jul 06 19:28:27.267434 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.267410 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84d69ccdc4-tv2kz"] Jul 06 19:28:27.270210 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.270183 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84d69ccdc4-tv2kz"] Jul 06 19:28:27.318642 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.318562 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/919dec9b-48ae-419f-b437-0627ccfb100a-registry-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:28:27.491891 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.491854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.578337 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.578259 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.621358 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.621327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:28:27.621496 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.621403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:28:27.624690 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.624508 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:28:27.624794 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.624687 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:28:27.636452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.636423 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4661f471-2b64-4e51-9361-f90b1dcc9910-metrics-certs\") pod \"network-metrics-daemon-xwq6w\" (UID: \"4661f471-2b64-4e51-9361-f90b1dcc9910\") " pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:28:27.637942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.637878 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d11926aa-a0cc-40d1-9c68-17a44704d1b6-original-pull-secret\") pod \"global-pull-secret-syncer-65kqw\" (UID: \"d11926aa-a0cc-40d1-9c68-17a44704d1b6\") " pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:28:27.650000 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.649961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.722727 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.722363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:28:27.725595 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.725569 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:28:27.736207 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.736033 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:28:27.746483 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.746431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7v8\" (UniqueName: \"kubernetes.io/projected/69d348dd-bd01-4286-98a6-3d5ba397a825-kube-api-access-lp7v8\") pod \"network-check-target-jqwdl\" (UID: \"69d348dd-bd01-4286-98a6-3d5ba397a825\") " pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:28:27.920209 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.920178 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdwjn\"" Jul 06 19:28:27.924392 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.924368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65kqw" Jul 06 19:28:27.927347 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.927329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:28:27.931450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.931429 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjmjf\"" Jul 06 19:28:27.939616 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:27.939075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwq6w" Jul 06 19:28:28.109462 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.109346 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65kqw"] Jul 06 19:28:28.114861 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:28.114823 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11926aa_a0cc_40d1_9c68_17a44704d1b6.slice/crio-eca7ee544a0d1979f303ca7a7ca146b497990c89ad5f2eff1f7c86fbee27f347 WatchSource:0}: Error finding container eca7ee544a0d1979f303ca7a7ca146b497990c89ad5f2eff1f7c86fbee27f347: Status 404 returned error can't find the container with id eca7ee544a0d1979f303ca7a7ca146b497990c89ad5f2eff1f7c86fbee27f347 Jul 06 19:28:28.141408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.141378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jqwdl"] Jul 06 19:28:28.144701 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:28.144674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d348dd_bd01_4286_98a6_3d5ba397a825.slice/crio-3737db87c28cd77bcef9fddf1be9e8d6919939c622a2d315ea07b622aa93589b WatchSource:0}: Error finding container 3737db87c28cd77bcef9fddf1be9e8d6919939c622a2d315ea07b622aa93589b: Status 404 returned error can't find the container with id 3737db87c28cd77bcef9fddf1be9e8d6919939c622a2d315ea07b622aa93589b Jul 06 19:28:28.219441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.219401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jqwdl" event={"ID":"69d348dd-bd01-4286-98a6-3d5ba397a825","Type":"ContainerStarted","Data":"3737db87c28cd77bcef9fddf1be9e8d6919939c622a2d315ea07b622aa93589b"} Jul 06 19:28:28.224396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.224345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65kqw" event={"ID":"d11926aa-a0cc-40d1-9c68-17a44704d1b6","Type":"ContainerStarted","Data":"eca7ee544a0d1979f303ca7a7ca146b497990c89ad5f2eff1f7c86fbee27f347"} Jul 06 19:28:28.337595 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.337491 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwq6w"] Jul 06 19:28:28.341952 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:28.341882 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4661f471_2b64_4e51_9361_f90b1dcc9910.slice/crio-8d8a751a024b588a107a263c6008b1d57a66ab72bd9bfb04d78d1ee8b6a59f7f WatchSource:0}: Error finding container 8d8a751a024b588a107a263c6008b1d57a66ab72bd9bfb04d78d1ee8b6a59f7f: Status 404 returned error can't find the container with id 8d8a751a024b588a107a263c6008b1d57a66ab72bd9bfb04d78d1ee8b6a59f7f Jul 06 19:28:28.809604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.809526 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919dec9b-48ae-419f-b437-0627ccfb100a" path="/var/lib/kubelet/pods/919dec9b-48ae-419f-b437-0627ccfb100a/volumes" Jul 06 19:28:28.842469 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.842439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:28.961552 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:28.961516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.045107 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.045076 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj"] Jul 06 19:28:29.051593 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.051478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:29.056180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.056158 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:28:29.056421 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.056404 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7vfxb\"" Jul 06 19:28:29.060408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.060308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.066001 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.065936 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj"] Jul 06 19:28:29.135255 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.135224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72010f0-fecf-4ae9-b1ae-a63dd896f581-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zxgkj\" (UID: \"a72010f0-fecf-4ae9-b1ae-a63dd896f581\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:29.161471 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.161049 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9228p"] Jul 06 19:28:29.164011 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.163938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:29.171934 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.171891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b8mcm\"" Jul 06 19:28:29.171934 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.171927 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:28:29.184671 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.184655 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:28:29.199986 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.199930 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9228p"] Jul 06 19:28:29.220361 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.220330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.230480 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.230452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwq6w" event={"ID":"4661f471-2b64-4e51-9361-f90b1dcc9910","Type":"ContainerStarted","Data":"8d8a751a024b588a107a263c6008b1d57a66ab72bd9bfb04d78d1ee8b6a59f7f"} Jul 06 19:28:29.235866 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.235832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5hc\" (UniqueName: \"kubernetes.io/projected/0061534e-5e79-49dc-917e-1036704ca034-kube-api-access-8d5hc\") pod \"downloads-685bfb5f64-9228p\" (UID: \"0061534e-5e79-49dc-917e-1036704ca034\") " pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:29.235982 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.235924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72010f0-fecf-4ae9-b1ae-a63dd896f581-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zxgkj\" (UID: \"a72010f0-fecf-4ae9-b1ae-a63dd896f581\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:29.239658 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.239276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a72010f0-fecf-4ae9-b1ae-a63dd896f581-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zxgkj\" (UID: \"a72010f0-fecf-4ae9-b1ae-a63dd896f581\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:29.293795 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.293762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.337237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.337163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5hc\" (UniqueName: \"kubernetes.io/projected/0061534e-5e79-49dc-917e-1036704ca034-kube-api-access-8d5hc\") pod \"downloads-685bfb5f64-9228p\" (UID: \"0061534e-5e79-49dc-917e-1036704ca034\") " pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:29.347042 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.346990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5hc\" (UniqueName: \"kubernetes.io/projected/0061534e-5e79-49dc-917e-1036704ca034-kube-api-access-8d5hc\") pod \"downloads-685bfb5f64-9228p\" (UID: \"0061534e-5e79-49dc-917e-1036704ca034\") " pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:29.369614 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.369551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:29.406677 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.406610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.476498 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.476470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:29.590259 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.590178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.772248 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.772198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.873398 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:29.873306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.147454 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.142795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj"] Jul 06 19:28:30.172138 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.172110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9228p"] Jul 06 19:28:30.177778 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.177699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.238606 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.237871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" event={"ID":"18ebce76-5361-4755-b835-30fe160b155e","Type":"ContainerStarted","Data":"9b7d95da4993d07e546eaf77e38f989895ba991840bef7175a6949aa275ed6e6"} Jul 06 19:28:30.240718 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.240647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" event={"ID":"a72010f0-fecf-4ae9-b1ae-a63dd896f581","Type":"ContainerStarted","Data":"6736390644f53d15482c27bf6f5a80532ea9ae0bb000149361f46ec4bb4f6211"} Jul 06 19:28:30.242641 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.242503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9228p" event={"ID":"0061534e-5e79-49dc-917e-1036704ca034","Type":"ContainerStarted","Data":"b2619082e8d5dff78d592ed7517886a9362be423b88b2422dbd893421dc3acae"} Jul 06 19:28:30.244416 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.244277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.257608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.257057 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-964d89bdd-9zlvf" podStartSLOduration=0.823855879 podStartE2EDuration="4.257043856s" totalImagesPullingTime="3.433187977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:30.256074467 +0000 UTC m=+67.909642093" watchObservedRunningTime="2026-07-06 19:28:30.257043856 +0000 UTC m=+67.910611454" Jul 06 19:28:30.391013 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.390926 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.490696 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.490622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.649376 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.649337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:30.653945 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.653921 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/de34a387-7a02-4be4-8c42-aba4d0989fb1-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-l7gvl\" (UID: \"de34a387-7a02-4be4-8c42-aba4d0989fb1\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:30.800845 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.800719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.900844 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.900813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.908927 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.908714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bksgq\"" Jul 06 19:28:30.915730 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:30.915709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" Jul 06 19:28:31.194225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:31.194101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:31.641003 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:31.640974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:31.742732 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:31.742702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:31.872612 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:31.872578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:31.941226 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:31.941130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:32.399354 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:32.399285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:32.501071 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:32.501021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:34.078489 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.078446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:34.081394 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.081350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/db58b7de-f080-402f-9ad4-e50c230c3e8f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-clz5w\" (UID: \"db58b7de-f080-402f-9ad4-e50c230c3e8f\") " pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:34.220711 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.220675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:34.250951 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.250921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmc6g\"" Jul 06 19:28:34.258057 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.258022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-clz5w" Jul 06 19:28:34.323499 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:34.323462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.142389 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.142324 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-clz5w"] Jul 06 19:28:35.158821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.158797 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl"] Jul 06 19:28:35.265997 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.265944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jqwdl" event={"ID":"69d348dd-bd01-4286-98a6-3d5ba397a825","Type":"ContainerStarted","Data":"e357ebecd1e8705d5e5773e47ea71a73a6288370af902a84bc1aaa310bb3576f"} Jul 06 19:28:35.266159 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.266054 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:28:35.267981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.267924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwq6w" event={"ID":"4661f471-2b64-4e51-9361-f90b1dcc9910","Type":"ContainerStarted","Data":"0e3f20fd9a7aadc26ba15deca7c3249fd917e0d5632802d421d40309e566e53a"} Jul 06 19:28:35.269873 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.269849 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" event={"ID":"af2b7195-c315-469b-b363-d44607a58d65","Type":"ContainerStarted","Data":"edeebe2ba2b7c88cabda6eab7dc152d7be0388f2384bf69266b6488c3fe85392"} Jul 06 19:28:35.270417 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.270393 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:35.271976 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.271959 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" Jul 06 19:28:35.289686 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.289641 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jqwdl" podStartSLOduration=66.481958701 podStartE2EDuration="1m13.28962251s" totalImagesPullingTime="6.807663809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:35.288799882 +0000 UTC m=+72.942367486" watchObservedRunningTime="2026-07-06 19:28:35.28962251 +0000 UTC m=+72.943190110" Jul 06 19:28:35.313873 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.313832 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-675c7796bf-gfq48" podStartSLOduration=0.924302975 podStartE2EDuration="9.313820052s" totalImagesPullingTime="8.389517077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:35.313255663 +0000 UTC m=+72.966823258" watchObservedRunningTime="2026-07-06 19:28:35.313820052 +0000 UTC m=+72.967387650" Jul 06 19:28:35.392396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.392366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.461801 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:35.461760 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb58b7de_f080_402f_9ad4_e50c230c3e8f.slice/crio-a11a1423b2fae7b9525787152f8c3e619daedb9a25212088631aa45205afc0b8 WatchSource:0}: Error finding container a11a1423b2fae7b9525787152f8c3e619daedb9a25212088631aa45205afc0b8: Status 404 returned error can't find the container with id a11a1423b2fae7b9525787152f8c3e619daedb9a25212088631aa45205afc0b8 Jul 06 19:28:35.462728 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:35.462588 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde34a387_7a02_4be4_8c42_aba4d0989fb1.slice/crio-0ab63486c45cf472798a65fc5795cef43347863e5274edffd7e2c69bbeb4bc25 WatchSource:0}: Error finding container 0ab63486c45cf472798a65fc5795cef43347863e5274edffd7e2c69bbeb4bc25: Status 404 returned error can't find the container with id 0ab63486c45cf472798a65fc5795cef43347863e5274edffd7e2c69bbeb4bc25 Jul 06 19:28:35.489936 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.489890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.870455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.870416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.964302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:35.964266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:36.278862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.278827 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65kqw" event={"ID":"d11926aa-a0cc-40d1-9c68-17a44704d1b6","Type":"ContainerStarted","Data":"896e71f4f8a9de59fa42341d7b72a12eaa15b1facab1b0d9ae461282a68bbb90"} Jul 06 19:28:36.284234 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.284202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwq6w" event={"ID":"4661f471-2b64-4e51-9361-f90b1dcc9910","Type":"ContainerStarted","Data":"8bf705d0cc52898eb278ea1d8b7b4250f7205890552378fadce17fc34739c266"} Jul 06 19:28:36.286634 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.286594 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" event={"ID":"a72010f0-fecf-4ae9-b1ae-a63dd896f581","Type":"ContainerStarted","Data":"9e443180ad57a30e9de4229dd4404b3145b0e2f75b0ec90a2680b98e6b684f81"} Jul 06 19:28:36.289258 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.287515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:36.290411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.290343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clz5w" event={"ID":"db58b7de-f080-402f-9ad4-e50c230c3e8f","Type":"ContainerStarted","Data":"a11a1423b2fae7b9525787152f8c3e619daedb9a25212088631aa45205afc0b8"} Jul 06 19:28:36.292540 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.292474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" event={"ID":"de34a387-7a02-4be4-8c42-aba4d0989fb1","Type":"ContainerStarted","Data":"0ab63486c45cf472798a65fc5795cef43347863e5274edffd7e2c69bbeb4bc25"} Jul 06 19:28:36.295451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.295325 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" Jul 06 19:28:36.297921 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.297849 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-65kqw" podStartSLOduration=66.922680494 podStartE2EDuration="1m14.297832864s" totalImagesPullingTime="7.37515237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:36.297013778 +0000 UTC m=+73.950581380" watchObservedRunningTime="2026-07-06 19:28:36.297832864 +0000 UTC m=+73.951400463" Jul 06 19:28:36.317581 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.316753 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zxgkj" podStartSLOduration=1.64907158 podStartE2EDuration="7.316735562s" totalImagesPullingTime="5.667663982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:36.315893573 +0000 UTC m=+73.969461172" watchObservedRunningTime="2026-07-06 19:28:36.316735562 +0000 UTC m=+73.970303162" Jul 06 19:28:36.344191 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.344071 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwq6w" podStartSLOduration=67.745493695 podStartE2EDuration="1m14.344051978s" totalImagesPullingTime="6.598558283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:36.343570878 +0000 UTC m=+73.997138478" watchObservedRunningTime="2026-07-06 19:28:36.344051978 +0000 UTC m=+73.997619577" Jul 06 19:28:36.528738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.528668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:36.629400 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:36.629288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.076971 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:37.076934 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.203013 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:37.202919 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.291100 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:37.291024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:38.038656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:38.038621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:38.137441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:38.137376 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.108806 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.108775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.309756 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.309665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clz5w" event={"ID":"db58b7de-f080-402f-9ad4-e50c230c3e8f","Type":"ContainerStarted","Data":"7540cd5eb0b7ba2340ca89b24706a9875a9d5159828623a4155fba7ac31060c2"} Jul 06 19:28:40.311165 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.311138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" event={"ID":"de34a387-7a02-4be4-8c42-aba4d0989fb1","Type":"ContainerStarted","Data":"1f7ca31d6ce050ae3484c1e7dca38ad8240ae9169f412aa5c5d5e7c306e92aee"} Jul 06 19:28:40.336096 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.336049 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-l7gvl" podStartSLOduration=37.826759488 podStartE2EDuration="42.336032794s" totalImagesPullingTime="4.509273306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:40.334415151 +0000 UTC m=+77.987982751" watchObservedRunningTime="2026-07-06 19:28:40.336032794 +0000 UTC m=+77.989600393" Jul 06 19:28:40.472060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.472026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.571952 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.571925 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.971324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:40.971274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.062806 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.062773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.284577 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.284485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.318353 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.318267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clz5w" event={"ID":"db58b7de-f080-402f-9ad4-e50c230c3e8f","Type":"ContainerStarted","Data":"7470192c77c0b5ab0760af798f6301657fe9cb3cf8ff3fd22bc4fde684ec335b"} Jul 06 19:28:41.518315 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.518252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.613800 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.613575 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h4p9t"] Jul 06 19:28:41.616145 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.616124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.619826 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.619739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:28:41.619826 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.619744 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:28:41.621973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.621424 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fxbq4\"" Jul 06 19:28:41.621973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.621627 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:28:41.621973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.621638 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:28:41.621973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.621821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:28:41.623847 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.623665 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:28:41.654675 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.654636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.741387 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-textfile\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-wtmp\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-metrics-client-ca\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-root\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mx4j\" (UniqueName: \"kubernetes.io/projected/d88ddcf6-b088-43d1-aff7-249563ad1093-kube-api-access-8mx4j\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-sys\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741882 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.741882 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.741632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.748837 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.748798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.826366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.826272 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.843086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-wtmp\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-metrics-client-ca\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-root\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mx4j\" (UniqueName: \"kubernetes.io/projected/d88ddcf6-b088-43d1-aff7-249563ad1093-kube-api-access-8mx4j\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-sys\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-wtmp\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-textfile\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:41.843351 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843475 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:28:41.843416 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls podName:d88ddcf6-b088-43d1-aff7-249563ad1093 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:42.343394026 +0000 UTC m=+79.996961618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls") pod "node-exporter-h4p9t" (UID: "d88ddcf6-b088-43d1-aff7-249563ad1093") : secret "node-exporter-tls" not found Jul 06 19:28:41.843775 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843568 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-root\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843775 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-textfile\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843814 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d88ddcf6-b088-43d1-aff7-249563ad1093-sys\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.843870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.843858 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-metrics-client-ca\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.844080 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.844050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.846390 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.846350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:41.864666 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:41.864561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mx4j\" (UniqueName: \"kubernetes.io/projected/d88ddcf6-b088-43d1-aff7-249563ad1093-kube-api-access-8mx4j\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:42.005242 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.005208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.347040 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.346927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:42.349700 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.349364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d88ddcf6-b088-43d1-aff7-249563ad1093-node-exporter-tls\") pod \"node-exporter-h4p9t\" (UID: \"d88ddcf6-b088-43d1-aff7-249563ad1093\") " pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:42.362560 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.362494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.528703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.528671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.531671 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.531644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4p9t" Jul 06 19:28:42.639388 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.639315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.744443 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.744149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.969996 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:42.969921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.469848 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:28:43.469805 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88ddcf6_b088_43d1_aff7_249563ad1093.slice/crio-cf2aa3c8921432854f63bb97bc67bb8fc95e5ae1c82bd2093ea93081d7415fbf WatchSource:0}: Error finding container cf2aa3c8921432854f63bb97bc67bb8fc95e5ae1c82bd2093ea93081d7415fbf: Status 404 returned error can't find the container with id cf2aa3c8921432854f63bb97bc67bb8fc95e5ae1c82bd2093ea93081d7415fbf Jul 06 19:28:44.333060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.333014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4p9t" event={"ID":"d88ddcf6-b088-43d1-aff7-249563ad1093","Type":"ContainerStarted","Data":"cf2aa3c8921432854f63bb97bc67bb8fc95e5ae1c82bd2093ea93081d7415fbf"} Jul 06 19:28:44.335357 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.335323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-clz5w" event={"ID":"db58b7de-f080-402f-9ad4-e50c230c3e8f","Type":"ContainerStarted","Data":"6936af08fa5665cb119d0ada7c2a6fa9461d3f315716758c5b25be8b7065375b"} Jul 06 19:28:44.366623 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.366564 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-clz5w" podStartSLOduration=34.464708339 podStartE2EDuration="42.366547639s" totalImagesPullingTime="7.9018393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:28:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:44.364824156 +0000 UTC m=+82.018391756" watchObservedRunningTime="2026-07-06 19:28:44.366547639 +0000 UTC m=+82.020115237" Jul 06 19:28:44.566570 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.566531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:44.645946 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.645861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:44.820638 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.820606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:44.919462 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:44.919378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:45.021657 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:45.021625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:45.109699 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:45.109665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:45.856943 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:45.856914 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:45.956670 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:45.956638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:46.301364 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:46.301328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:46.397253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:46.397190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.054776 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:47.054740 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.153410 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:47.153305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.253968 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:47.253932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.937875 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:47.937842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:48.229365 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.229338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf4c59977-qr592" Jul 06 19:28:48.260912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.260869 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:48.340882 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.340854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:48.440876 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.440836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:48.541469 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.541359 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:48.642191 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:48.642159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:49.225443 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:49.225390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:49.320912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:49.320863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:52.368125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.368089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9228p" event={"ID":"0061534e-5e79-49dc-917e-1036704ca034","Type":"ContainerStarted","Data":"c76dad664c117cb327bed1bbbd0e4e5001fa0bce8b695cbb14c9970b9d0425bf"} Jul 06 19:28:52.368609 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.368275 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:52.370225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.370135 2577 generic.go:356] "Generic (PLEG): container finished" podID="d88ddcf6-b088-43d1-aff7-249563ad1093" containerID="56f58070678ef49fc2d0a07aed990e31cb25c49f4a785cb8897f7bf65ad8a231" exitCode=0 Jul 06 19:28:52.370225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.370169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4p9t" event={"ID":"d88ddcf6-b088-43d1-aff7-249563ad1093","Type":"ContainerDied","Data":"56f58070678ef49fc2d0a07aed990e31cb25c49f4a785cb8897f7bf65ad8a231"} Jul 06 19:28:52.384175 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.384152 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9228p" Jul 06 19:28:52.391689 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:52.391640 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9228p" podStartSLOduration=2.167892567 podStartE2EDuration="23.391622276s" totalImagesPullingTime="21.223729709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:52.388947853 +0000 UTC m=+90.042515455" watchObservedRunningTime="2026-07-06 19:28:52.391622276 +0000 UTC m=+90.045189875" Jul 06 19:28:53.377287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:53.377249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4p9t" event={"ID":"d88ddcf6-b088-43d1-aff7-249563ad1093","Type":"ContainerStarted","Data":"a13fbacbe18b3fae0354e1737d50f9b87c3813de5c696440bbac11fc598819de"} Jul 06 19:28:53.377774 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:53.377311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4p9t" event={"ID":"d88ddcf6-b088-43d1-aff7-249563ad1093","Type":"ContainerStarted","Data":"baaaa1854f67c2a1b5d5d18ad02474cb57fda44a91d2b7eb46e4f2d1d3e5d6f5"} Jul 06 19:28:53.404345 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:53.404290 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h4p9t" podStartSLOduration=3.8694262999999998 podStartE2EDuration="12.404272843s" totalImagesPullingTime="8.432752906s" totalInitContainerRuntime="102.093637ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:53.400910874 +0000 UTC m=+91.054478470" watchObservedRunningTime="2026-07-06 19:28:53.404272843 +0000 UTC m=+91.057840442" Jul 06 19:28:54.298340 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:54.298302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:54.379919 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:54.379875 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:54.615893 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:54.615810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:54.681231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:54.681194 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:55.129745 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:55.129709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:55.189838 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:55.189805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:55.593219 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:55.593188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:55.695017 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:55.694985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:56.210894 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:56.210859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:56.265863 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:56.265831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:58.811649 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:58.811600 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:58.879619 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:28:58.879589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.411151 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:04.411111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.511417 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:04.511384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.572277 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:04.572252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:06.295026 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:06.294997 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jqwdl" Jul 06 19:29:10.422059 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:10.422027 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:10.477890 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:10.477862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:18.909584 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:18.909540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:18.975480 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:18.975448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:52.546124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:52.546091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:53.111730 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:29:53.111699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:29.428117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:29.428038 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:29.528694 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:29.528665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:30.957793 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:30.957762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:31.049718 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:31.049685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:40.500360 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:40.500297 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.893657 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:51.893625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.994377 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:51.994348 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:52.095638 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:30:52.095607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.637604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:02.637562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.702155 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:02.702128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:12.125846 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:12.125805 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:46.250963 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:46.250876 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:46.309066 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:46.309045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:47.573131 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:47.573099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:47.673891 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:47.673857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:47.738199 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:47.738169 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:48.296557 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:48.296522 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:48.363498 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:48.363469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:58.183500 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:58.183467 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:58.283601 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:58.283568 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:58.369277 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:31:58.369250 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:09.096710 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:09.096674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:10.615235 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:10.615201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:10.709734 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:10.709705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.708860 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:22.708833 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:32:43.341581 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:43.341543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:44.329179 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.329146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt"] Jul 06 19:32:44.331211 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.331196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.334472 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.334448 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:32:44.335701 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.335673 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:32:44.335810 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.335726 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mcprt\"" Jul 06 19:32:44.346103 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.346080 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt"] Jul 06 19:32:44.412548 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.412510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.412659 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.412570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrnp\" (UniqueName: \"kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.412659 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.412604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.513144 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.513112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrnp\" (UniqueName: \"kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.513246 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.513147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.513246 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.513186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.513539 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.513523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.513580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.513544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.522803 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.522779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrnp\" (UniqueName: \"kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.537008 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.536988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:44.640735 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.640670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:32:44.762695 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.762661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt"] Jul 06 19:32:44.765554 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:32:44.765526 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd54d32_f0b3_4104_a3a6_49a59575b1db.slice/crio-52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44 WatchSource:0}: Error finding container 52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44: Status 404 returned error can't find the container with id 52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44 Jul 06 19:32:44.767455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:44.767432 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:32:45.129701 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:45.129666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" event={"ID":"7fd54d32-f0b3-4104-a3a6-49a59575b1db","Type":"ContainerStarted","Data":"52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44"} Jul 06 19:32:52.159226 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:52.159191 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerID="a6e733ac450a3ec7e1bc38d576349097958b94d3539700c1904d411f8a914eda" exitCode=0 Jul 06 19:32:52.159568 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:52.159253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" event={"ID":"7fd54d32-f0b3-4104-a3a6-49a59575b1db","Type":"ContainerDied","Data":"a6e733ac450a3ec7e1bc38d576349097958b94d3539700c1904d411f8a914eda"} Jul 06 19:32:55.171524 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:55.171491 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerID="e8ccca71efe3288007b87ecd064169a12388ad77e4c166802be1e9954fc4bf78" exitCode=0 Jul 06 19:32:55.172010 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:32:55.171541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" event={"ID":"7fd54d32-f0b3-4104-a3a6-49a59575b1db","Type":"ContainerDied","Data":"e8ccca71efe3288007b87ecd064169a12388ad77e4c166802be1e9954fc4bf78"} Jul 06 19:33:04.206129 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:04.206041 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerID="c25d6b22bc15befd1726239cf779b52923b20b88510afa40aa1dcc052b561aeb" exitCode=0 Jul 06 19:33:04.206129 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:04.206085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" event={"ID":"7fd54d32-f0b3-4104-a3a6-49a59575b1db","Type":"ContainerDied","Data":"c25d6b22bc15befd1726239cf779b52923b20b88510afa40aa1dcc052b561aeb"} Jul 06 19:33:05.328303 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.328278 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:33:05.376081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.376058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle\") pod \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " Jul 06 19:33:05.376199 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.376140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbrnp\" (UniqueName: \"kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp\") pod \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " Jul 06 19:33:05.376265 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.376203 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util\") pod \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\" (UID: \"7fd54d32-f0b3-4104-a3a6-49a59575b1db\") " Jul 06 19:33:05.376661 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.376638 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle" (OuterVolumeSpecName: "bundle") pod "7fd54d32-f0b3-4104-a3a6-49a59575b1db" (UID: "7fd54d32-f0b3-4104-a3a6-49a59575b1db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:05.378314 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.378283 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp" (OuterVolumeSpecName: "kube-api-access-kbrnp") pod "7fd54d32-f0b3-4104-a3a6-49a59575b1db" (UID: "7fd54d32-f0b3-4104-a3a6-49a59575b1db"). InnerVolumeSpecName "kube-api-access-kbrnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:05.380942 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.380916 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util" (OuterVolumeSpecName: "util") pod "7fd54d32-f0b3-4104-a3a6-49a59575b1db" (UID: "7fd54d32-f0b3-4104-a3a6-49a59575b1db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:05.476748 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.476725 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrnp\" (UniqueName: \"kubernetes.io/projected/7fd54d32-f0b3-4104-a3a6-49a59575b1db-kube-api-access-kbrnp\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:05.476840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.476748 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:05.476840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:05.476758 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd54d32-f0b3-4104-a3a6-49a59575b1db-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:06.215221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:06.215190 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" Jul 06 19:33:06.215221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:06.215203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754088jnt" event={"ID":"7fd54d32-f0b3-4104-a3a6-49a59575b1db","Type":"ContainerDied","Data":"52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44"} Jul 06 19:33:06.215412 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:06.215246 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52851319581dffe12e980045bbebf05d7cea86b0863989562b0924837c39ba44" Jul 06 19:33:10.450559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:10.450525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:11.255621 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:11.255588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:11.987120 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:11.987079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.089150 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.089106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:12.782200 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782164 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4"] Jul 06 19:33:12.782408 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782397 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="extract" Jul 06 19:33:12.782450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782410 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="extract" Jul 06 19:33:12.782450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782420 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="util" Jul 06 19:33:12.782450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782426 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="util" Jul 06 19:33:12.782450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782433 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="pull" Jul 06 19:33:12.782450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782440 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="pull" Jul 06 19:33:12.782596 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.782493 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd54d32-f0b3-4104-a3a6-49a59575b1db" containerName="extract" Jul 06 19:33:12.785824 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.785807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.790115 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.790098 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:33:12.790235 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.790184 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rf7rg\"" Jul 06 19:33:12.790235 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.790198 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:33:12.808537 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.808513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4"] Jul 06 19:33:12.823797 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.823777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/679cd297-34be-4235-9aaf-137ce18ed5f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.823928 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.823840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdck\" (UniqueName: \"kubernetes.io/projected/679cd297-34be-4235-9aaf-137ce18ed5f3-kube-api-access-8qdck\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.924683 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.924657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdck\" (UniqueName: \"kubernetes.io/projected/679cd297-34be-4235-9aaf-137ce18ed5f3-kube-api-access-8qdck\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.924842 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.924728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/679cd297-34be-4235-9aaf-137ce18ed5f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.925162 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.925143 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/679cd297-34be-4235-9aaf-137ce18ed5f3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:12.934029 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:12.934007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdck\" (UniqueName: \"kubernetes.io/projected/679cd297-34be-4235-9aaf-137ce18ed5f3-kube-api-access-8qdck\") pod \"cert-manager-operator-controller-manager-55c444bf86-k2dq4\" (UID: \"679cd297-34be-4235-9aaf-137ce18ed5f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:13.094448 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:13.094360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" Jul 06 19:33:13.221667 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:13.221643 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4"] Jul 06 19:33:13.224002 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:13.223975 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679cd297_34be_4235_9aaf_137ce18ed5f3.slice/crio-7482c1e7902d25b514441eff63262c73b867ede0a58328a6ddfc6bf65d4fd409 WatchSource:0}: Error finding container 7482c1e7902d25b514441eff63262c73b867ede0a58328a6ddfc6bf65d4fd409: Status 404 returned error can't find the container with id 7482c1e7902d25b514441eff63262c73b867ede0a58328a6ddfc6bf65d4fd409 Jul 06 19:33:13.241024 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:13.240985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" event={"ID":"679cd297-34be-4235-9aaf-137ce18ed5f3","Type":"ContainerStarted","Data":"7482c1e7902d25b514441eff63262c73b867ede0a58328a6ddfc6bf65d4fd409"} Jul 06 19:33:14.258500 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:14.258465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.362185 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:14.362153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:18.874612 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:18.874580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:18.937235 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:18.933463 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.206497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.206455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.273063 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.273026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" event={"ID":"679cd297-34be-4235-9aaf-137ce18ed5f3","Type":"ContainerStarted","Data":"2f88ce18249dcf1e6d0788da98e04b9aedad5cf86349c2cd20b20d490872347c"} Jul 06 19:33:21.302231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.302177 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-k2dq4" podStartSLOduration=2.036700046 podStartE2EDuration="9.302140307s" totalImagesPullingTime="7.265440261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:21.300510053 +0000 UTC m=+358.954077651" watchObservedRunningTime="2026-07-06 19:33:21.302140307 +0000 UTC m=+358.955707906" Jul 06 19:33:21.302978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.302949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.773295 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.773256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.907086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:21.907052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:23.404186 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.404155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:23.499485 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.499450 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7"] Jul 06 19:33:23.502831 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.502815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.506026 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.506009 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:33:23.507110 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.507090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:33:23.507198 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.507119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mcprt\"" Jul 06 19:33:23.515888 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.515863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7"] Jul 06 19:33:23.555058 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.555025 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9"] Jul 06 19:33:23.558010 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.557994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.561019 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.560990 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:33:23.561190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.561171 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v89pc\"" Jul 06 19:33:23.561349 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.561332 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:33:23.571874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.571853 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9"] Jul 06 19:33:23.601025 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.600997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.601156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.601030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s676k\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-kube-api-access-s676k\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.601156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.601067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrk5g\" (UniqueName: \"kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.601156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.601124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.601156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.601154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.648776 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.648745 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:23.702189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.702189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s676k\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-kube-api-access-s676k\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.702189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrk5g\" (UniqueName: \"kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.702410 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.702410 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.702618 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.702689 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.702601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.711608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.711587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrk5g\" (UniqueName: \"kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.711730 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.711697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.711857 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.711839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s676k\" (UniqueName: \"kubernetes.io/projected/9edc62d7-f029-4556-aab7-2a56c8198f4d-kube-api-access-s676k\") pod \"cert-manager-cainjector-559cdd8c87-2d7w9\" (UID: \"9edc62d7-f029-4556-aab7-2a56c8198f4d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.811581 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.811545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:23.866766 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.866733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" Jul 06 19:33:23.947455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:23.947428 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7"] Jul 06 19:33:23.948386 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:23.948353 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7b91ee_7413_45f9_bf96_1500fa583d77.slice/crio-21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7 WatchSource:0}: Error finding container 21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7: Status 404 returned error can't find the container with id 21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7 Jul 06 19:33:24.009066 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:24.009043 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9"] Jul 06 19:33:24.011322 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:24.011297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9edc62d7_f029_4556_aab7_2a56c8198f4d.slice/crio-33a0bf0e5924ba45b54ac69ac2096c928570d56a9e654c0d583ccccf01a78295 WatchSource:0}: Error finding container 33a0bf0e5924ba45b54ac69ac2096c928570d56a9e654c0d583ccccf01a78295: Status 404 returned error can't find the container with id 33a0bf0e5924ba45b54ac69ac2096c928570d56a9e654c0d583ccccf01a78295 Jul 06 19:33:24.289776 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:24.289744 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerID="bb83f62e43127975147668735b5c80dd22258883a5e45d29e4264463e9211c12" exitCode=0 Jul 06 19:33:24.289958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:24.289836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" event={"ID":"3a7b91ee-7413-45f9-bf96-1500fa583d77","Type":"ContainerDied","Data":"bb83f62e43127975147668735b5c80dd22258883a5e45d29e4264463e9211c12"} Jul 06 19:33:24.289958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:24.289890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" event={"ID":"3a7b91ee-7413-45f9-bf96-1500fa583d77","Type":"ContainerStarted","Data":"21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7"} Jul 06 19:33:24.291012 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:24.290985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" event={"ID":"9edc62d7-f029-4556-aab7-2a56c8198f4d","Type":"ContainerStarted","Data":"33a0bf0e5924ba45b54ac69ac2096c928570d56a9e654c0d583ccccf01a78295"} Jul 06 19:33:28.315754 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.315719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" event={"ID":"9edc62d7-f029-4556-aab7-2a56c8198f4d","Type":"ContainerStarted","Data":"9b7a0b05254b441a909ee1bf065ba70022a611b2731cbb5c0b15774818c96a59"} Jul 06 19:33:28.317417 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.317393 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerID="71640d4e90b4ed6c4161574cdf8455cc1e3c19cf3208e9065f44e5ea3b937a15" exitCode=0 Jul 06 19:33:28.317534 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.317459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" event={"ID":"3a7b91ee-7413-45f9-bf96-1500fa583d77","Type":"ContainerDied","Data":"71640d4e90b4ed6c4161574cdf8455cc1e3c19cf3208e9065f44e5ea3b937a15"} Jul 06 19:33:28.369102 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.369057 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d7w9" podStartSLOduration=2.046388293 podStartE2EDuration="5.369041408s" totalImagesPullingTime="3.322653115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:28.335563452 +0000 UTC m=+365.989131063" watchObservedRunningTime="2026-07-06 19:33:28.369041408 +0000 UTC m=+366.022609007" Jul 06 19:33:28.458090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.458067 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:28.546765 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:28.546737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:29.323461 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:29.323424 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerID="dcf9699dacab8ea919e099e71d477740d06a544ecca1191c9a68a2b947413c7c" exitCode=0 Jul 06 19:33:29.323862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:29.323505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" event={"ID":"3a7b91ee-7413-45f9-bf96-1500fa583d77","Type":"ContainerDied","Data":"dcf9699dacab8ea919e099e71d477740d06a544ecca1191c9a68a2b947413c7c"} Jul 06 19:33:30.445536 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.445508 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:30.561460 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.561418 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util\") pod \"3a7b91ee-7413-45f9-bf96-1500fa583d77\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " Jul 06 19:33:30.561634 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.561470 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle\") pod \"3a7b91ee-7413-45f9-bf96-1500fa583d77\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " Jul 06 19:33:30.561634 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.561502 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrk5g\" (UniqueName: \"kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g\") pod \"3a7b91ee-7413-45f9-bf96-1500fa583d77\" (UID: \"3a7b91ee-7413-45f9-bf96-1500fa583d77\") " Jul 06 19:33:30.561928 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.561888 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle" (OuterVolumeSpecName: "bundle") pod "3a7b91ee-7413-45f9-bf96-1500fa583d77" (UID: "3a7b91ee-7413-45f9-bf96-1500fa583d77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:30.563545 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.563524 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g" (OuterVolumeSpecName: "kube-api-access-lrk5g") pod "3a7b91ee-7413-45f9-bf96-1500fa583d77" (UID: "3a7b91ee-7413-45f9-bf96-1500fa583d77"). InnerVolumeSpecName "kube-api-access-lrk5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:30.566077 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.566054 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util" (OuterVolumeSpecName: "util") pod "3a7b91ee-7413-45f9-bf96-1500fa583d77" (UID: "3a7b91ee-7413-45f9-bf96-1500fa583d77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:30.662276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.662196 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrk5g\" (UniqueName: \"kubernetes.io/projected/3a7b91ee-7413-45f9-bf96-1500fa583d77-kube-api-access-lrk5g\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:30.662276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.662226 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:30.662276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:30.662237 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a7b91ee-7413-45f9-bf96-1500fa583d77-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:31.333590 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:31.333550 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" event={"ID":"3a7b91ee-7413-45f9-bf96-1500fa583d77","Type":"ContainerDied","Data":"21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7"} Jul 06 19:33:31.333752 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:31.333604 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f43d6204887ad075eade903d002eae9741660693c0e349d2374b5ce10a21a7" Jul 06 19:33:31.333752 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:31.333573 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzktv7" Jul 06 19:33:31.843076 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:31.843042 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:34.967607 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:34.967576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:35.124031 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:35.123995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:41.163346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.163309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:41.266230 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.266204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:41.791571 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791537 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mlhnk"] Jul 06 19:33:41.791830 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791817 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="pull" Jul 06 19:33:41.791874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791832 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="pull" Jul 06 19:33:41.791874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791846 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="extract" Jul 06 19:33:41.791874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791851 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="extract" Jul 06 19:33:41.791874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791864 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="util" Jul 06 19:33:41.791874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791869 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="util" Jul 06 19:33:41.792042 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.791935 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7b91ee-7413-45f9-bf96-1500fa583d77" containerName="extract" Jul 06 19:33:41.794092 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.794078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:41.798223 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.798207 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8szvv\"" Jul 06 19:33:41.807328 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.807300 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mlhnk"] Jul 06 19:33:41.947394 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.947364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:41.947394 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:41.947397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d278d\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-kube-api-access-d278d\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.048011 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.047916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.048011 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.047971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d278d\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-kube-api-access-d278d\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.057243 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.057216 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.057333 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.057318 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d278d\" (UniqueName: \"kubernetes.io/projected/d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa-kube-api-access-d278d\") pod \"cert-manager-55b4d954d5-mlhnk\" (UID: \"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa\") " pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.102172 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.102148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mlhnk" Jul 06 19:33:42.225736 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.225713 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mlhnk"] Jul 06 19:33:42.227608 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:42.227561 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2cd7d9f_7cba_4277_90c2_c1e8dc49fbfa.slice/crio-9af1ef6b49d1373fad46cc99f314a6b50a3281eb9a4a761f18e2b475d5a13ccd WatchSource:0}: Error finding container 9af1ef6b49d1373fad46cc99f314a6b50a3281eb9a4a761f18e2b475d5a13ccd: Status 404 returned error can't find the container with id 9af1ef6b49d1373fad46cc99f314a6b50a3281eb9a4a761f18e2b475d5a13ccd Jul 06 19:33:42.375547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.375468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mlhnk" event={"ID":"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa","Type":"ContainerStarted","Data":"3c2c94c360d86d526f4656e06ccad9196c605aab5b7066a925829cd216fb9fa6"} Jul 06 19:33:42.375547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.375509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mlhnk" event={"ID":"d2cd7d9f-7cba-4277-90c2-c1e8dc49fbfa","Type":"ContainerStarted","Data":"9af1ef6b49d1373fad46cc99f314a6b50a3281eb9a4a761f18e2b475d5a13ccd"} Jul 06 19:33:42.394172 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:42.394129 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mlhnk" podStartSLOduration=1.394116131 podStartE2EDuration="1.394116131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:42.392991432 +0000 UTC m=+380.046559033" watchObservedRunningTime="2026-07-06 19:33:42.394116131 +0000 UTC m=+380.047683724" Jul 06 19:33:43.005060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.005028 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66"] Jul 06 19:33:43.007096 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.007080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.009968 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.009950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:33:43.011135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.011115 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:33:43.011240 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.011145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mcprt\"" Jul 06 19:33:43.017912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.017880 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66"] Jul 06 19:33:43.157864 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.157833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.158023 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.157873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.158023 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.157973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g2v\" (UniqueName: \"kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.259253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.259196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g2v\" (UniqueName: \"kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.259253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.259230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.259593 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.259255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.259657 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.259636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.259703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.259649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.271071 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.271040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g2v\" (UniqueName: \"kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.315971 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.315947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:43.318555 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.318537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:43.441426 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:43.441402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66"] Jul 06 19:33:43.443516 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:43.443486 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3605416_c5cd_49b3_adb6_99df55058e6e.slice/crio-a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf WatchSource:0}: Error finding container a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf: Status 404 returned error can't find the container with id a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf Jul 06 19:33:44.385246 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:44.385216 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerID="569d9fdf20092125939f01faadb2d833fd02337caa36f9bb284354d69a099c71" exitCode=0 Jul 06 19:33:44.385626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:44.385253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" event={"ID":"b3605416-c5cd-49b3-adb6-99df55058e6e","Type":"ContainerDied","Data":"569d9fdf20092125939f01faadb2d833fd02337caa36f9bb284354d69a099c71"} Jul 06 19:33:44.385626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:44.385291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" event={"ID":"b3605416-c5cd-49b3-adb6-99df55058e6e","Type":"ContainerStarted","Data":"a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf"} Jul 06 19:33:45.391184 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:45.391095 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerID="aff873d8fc41cd61e659254c534f27c8326ca7629635daba80d3802594a77866" exitCode=0 Jul 06 19:33:45.391579 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:45.391181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" event={"ID":"b3605416-c5cd-49b3-adb6-99df55058e6e","Type":"ContainerDied","Data":"aff873d8fc41cd61e659254c534f27c8326ca7629635daba80d3802594a77866"} Jul 06 19:33:46.397690 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:46.397657 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerID="92b22e316fa5c6634650118233e42834169005ab2504f1083a68a47ee69e3811" exitCode=0 Jul 06 19:33:46.398084 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:46.397727 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" event={"ID":"b3605416-c5cd-49b3-adb6-99df55058e6e","Type":"ContainerDied","Data":"92b22e316fa5c6634650118233e42834169005ab2504f1083a68a47ee69e3811"} Jul 06 19:33:47.069724 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.069682 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:47.137700 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.137667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:47.523892 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.523867 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:47.692507 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.692431 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65g2v\" (UniqueName: \"kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v\") pod \"b3605416-c5cd-49b3-adb6-99df55058e6e\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " Jul 06 19:33:47.692507 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.692476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util\") pod \"b3605416-c5cd-49b3-adb6-99df55058e6e\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " Jul 06 19:33:47.692696 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.692556 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle\") pod \"b3605416-c5cd-49b3-adb6-99df55058e6e\" (UID: \"b3605416-c5cd-49b3-adb6-99df55058e6e\") " Jul 06 19:33:47.693242 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.693214 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle" (OuterVolumeSpecName: "bundle") pod "b3605416-c5cd-49b3-adb6-99df55058e6e" (UID: "b3605416-c5cd-49b3-adb6-99df55058e6e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:47.694556 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.694531 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v" (OuterVolumeSpecName: "kube-api-access-65g2v") pod "b3605416-c5cd-49b3-adb6-99df55058e6e" (UID: "b3605416-c5cd-49b3-adb6-99df55058e6e"). InnerVolumeSpecName "kube-api-access-65g2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:47.699866 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.699840 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util" (OuterVolumeSpecName: "util") pod "b3605416-c5cd-49b3-adb6-99df55058e6e" (UID: "b3605416-c5cd-49b3-adb6-99df55058e6e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:33:47.793564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.793531 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g2v\" (UniqueName: \"kubernetes.io/projected/b3605416-c5cd-49b3-adb6-99df55058e6e-kube-api-access-65g2v\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:47.793564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.793558 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:47.793564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:47.793568 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3605416-c5cd-49b3-adb6-99df55058e6e-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:33:48.407345 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:48.407318 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" Jul 06 19:33:48.407345 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:48.407330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afttp66" event={"ID":"b3605416-c5cd-49b3-adb6-99df55058e6e","Type":"ContainerDied","Data":"a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf"} Jul 06 19:33:48.407541 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:48.407370 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97c8cfe860915a55b1d5b93b944f2f28ca46918d7a7e14b81904ade81f0a3bf" Jul 06 19:33:54.951643 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:54.951612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:55.046197 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.046167 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:55.544137 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.544104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:55.838821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.838748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:55.938978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.938948 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:55.994874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.994846 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp"] Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995111 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="util" Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995122 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="util" Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995131 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="extract" Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995137 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="extract" Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995147 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="pull" Jul 06 19:33:55.995189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995153 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="pull" Jul 06 19:33:55.995368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:55.995203 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3605416-c5cd-49b3-adb6-99df55058e6e" containerName="extract" Jul 06 19:33:56.000552 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.000535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.004180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.004161 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:33:56.004450 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.004434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:33:56.004574 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.004550 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-csv8h\"" Jul 06 19:33:56.004712 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.004697 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:33:56.004972 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.004957 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:33:56.015436 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.015417 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp"] Jul 06 19:33:56.144846 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.144763 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.156891 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.156864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-manifests\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.156981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.156934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwg27\" (UniqueName: \"kubernetes.io/projected/0604ee50-a8e0-44d2-bc77-eae76ed81f19-kube-api-access-wwg27\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.156981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.156962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.157077 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.156980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.157130 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.157099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-tmp\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.239805 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.239782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.258320 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwg27\" (UniqueName: \"kubernetes.io/projected/0604ee50-a8e0-44d2-bc77-eae76ed81f19-kube-api-access-wwg27\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.258418 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.258418 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.258531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-tmp\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.258531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-manifests\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.258823 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.258803 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-manifests\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.260580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.260557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604ee50-a8e0-44d2-bc77-eae76ed81f19-tmp\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.260686 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.260661 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.260820 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.260802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604ee50-a8e0-44d2-bc77-eae76ed81f19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.267611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.267593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwg27\" (UniqueName: \"kubernetes.io/projected/0604ee50-a8e0-44d2-bc77-eae76ed81f19-kube-api-access-wwg27\") pod \"opendatahub-operator-controller-manager-57b585888c-b9qqp\" (UID: \"0604ee50-a8e0-44d2-bc77-eae76ed81f19\") " pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.310608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.310581 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:33:56.346684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.346661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.444578 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.444497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.455441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.455382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp"] Jul 06 19:33:56.457783 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:33:56.457758 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0604ee50_a8e0_44d2_bc77_eae76ed81f19.slice/crio-3808fbb68fe7025265fca5b567dd5f2b440cfbf79a08a1d06918701fa478bea8 WatchSource:0}: Error finding container 3808fbb68fe7025265fca5b567dd5f2b440cfbf79a08a1d06918701fa478bea8: Status 404 returned error can't find the container with id 3808fbb68fe7025265fca5b567dd5f2b440cfbf79a08a1d06918701fa478bea8 Jul 06 19:33:56.546564 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.546534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.648936 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.648911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:56.746991 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:56.746967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:57.444740 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:57.444705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" event={"ID":"0604ee50-a8e0-44d2-bc77-eae76ed81f19","Type":"ContainerStarted","Data":"3808fbb68fe7025265fca5b567dd5f2b440cfbf79a08a1d06918701fa478bea8"} Jul 06 19:33:59.456862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:33:59.456830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" event={"ID":"0604ee50-a8e0-44d2-bc77-eae76ed81f19","Type":"ContainerStarted","Data":"46e249e79ce8a674bc2a33e9bfae930befe4e9ffb6ca26c17b2cee21d5fc2318"} Jul 06 19:34:00.462815 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:00.462776 2577 generic.go:356] "Generic (PLEG): container finished" podID="0604ee50-a8e0-44d2-bc77-eae76ed81f19" containerID="46e249e79ce8a674bc2a33e9bfae930befe4e9ffb6ca26c17b2cee21d5fc2318" exitCode=0 Jul 06 19:34:00.463168 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:00.462855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" event={"ID":"0604ee50-a8e0-44d2-bc77-eae76ed81f19","Type":"ContainerDied","Data":"46e249e79ce8a674bc2a33e9bfae930befe4e9ffb6ca26c17b2cee21d5fc2318"} Jul 06 19:34:01.469342 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:01.469300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" event={"ID":"0604ee50-a8e0-44d2-bc77-eae76ed81f19","Type":"ContainerStarted","Data":"7b7dd5c0893648a0c2cf4adc23e3da0e009c70d4352ae16c235b1aca8e778dd0"} Jul 06 19:34:01.469738 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:01.469446 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:34:01.494017 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:01.493946 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" podStartSLOduration=2.114000268 podStartE2EDuration="6.493923005s" totalImagesPullingTime="3.352436215s" totalInitContainerRuntime="1.027486522s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:01.492553478 +0000 UTC m=+399.146121077" watchObservedRunningTime="2026-07-06 19:34:01.493923005 +0000 UTC m=+399.147490607" Jul 06 19:34:07.319515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:07.319483 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:07.438382 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:07.438344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:12.475709 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:12.475676 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57b585888c-b9qqp" Jul 06 19:34:14.635361 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:14.635327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:14.807098 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:14.807066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.270604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.270568 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.383432 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.383398 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58f6467984-49ldb"] Jul 06 19:34:15.391832 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.391803 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.399214 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.399189 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:34:15.400499 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.400476 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:34:15.400615 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.400497 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jpdf2\"" Jul 06 19:34:15.400615 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.400514 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:34:15.400615 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.400529 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:34:15.416131 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.416104 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58f6467984-49ldb"] Jul 06 19:34:15.492562 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.492517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.492767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.492591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9s7z\" (UniqueName: \"kubernetes.io/projected/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-kube-api-access-l9s7z\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.492767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.492649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tmp\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.593574 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.593485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tmp\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.593727 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.593578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.593727 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.593627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s7z\" (UniqueName: \"kubernetes.io/projected/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-kube-api-access-l9s7z\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.593808 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:34:15.593746 2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 06 19:34:15.593854 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:34:15.593826 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs podName:19a5ba3d-428d-4fd0-82e3-f45a8f81901c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:16.093804857 +0000 UTC m=+413.747372450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs") pod "kube-auth-proxy-58f6467984-49ldb" (UID: "19a5ba3d-428d-4fd0-82e3-f45a8f81901c") : secret "kube-auth-proxy-tls" not found Jul 06 19:34:15.595838 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.595806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tmp\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.612609 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.612583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s7z\" (UniqueName: \"kubernetes.io/projected/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-kube-api-access-l9s7z\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:15.737366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.737327 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.852577 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.852491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:15.959570 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:15.959530 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.065611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.065576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.097956 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.097910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:16.100294 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.100267 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19a5ba3d-428d-4fd0-82e3-f45a8f81901c-tls-certs\") pod \"kube-auth-proxy-58f6467984-49ldb\" (UID: \"19a5ba3d-428d-4fd0-82e3-f45a8f81901c\") " pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:16.302069 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.302034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" Jul 06 19:34:16.306423 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.306111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.450968 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.450942 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58f6467984-49ldb"] Jul 06 19:34:16.452795 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:34:16.452762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a5ba3d_428d_4fd0_82e3_f45a8f81901c.slice/crio-0da593f575151a29ccc0e990acf62738039ca95d9db1a32abe4a1c28e7353a15 WatchSource:0}: Error finding container 0da593f575151a29ccc0e990acf62738039ca95d9db1a32abe4a1c28e7353a15: Status 404 returned error can't find the container with id 0da593f575151a29ccc0e990acf62738039ca95d9db1a32abe4a1c28e7353a15 Jul 06 19:34:16.533393 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.533356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" event={"ID":"19a5ba3d-428d-4fd0-82e3-f45a8f81901c","Type":"ContainerStarted","Data":"0da593f575151a29ccc0e990acf62738039ca95d9db1a32abe4a1c28e7353a15"} Jul 06 19:34:16.798780 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.798741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:16.992759 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:16.992722 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.158508 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.158421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.318801 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.318772 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:17.322123 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.322101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.325471 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.325446 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:34:17.325471 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.325456 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n8d78\"" Jul 06 19:34:17.326324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.326303 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:34:17.326626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.326609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:34:17.328054 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.328037 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:34:17.337357 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.337335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.363389 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.358192 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:17.408668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.408668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.408860 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.408860 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.408860 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.408860 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgs2\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.409092 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.408867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.446679 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.446650 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.509436 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509605 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509674 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509725 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509725 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgs2\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.509840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.509773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.510725 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.510645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.512576 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.512552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.512701 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.512556 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.512701 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.512632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.512701 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.512640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.528915 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.528872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgs2\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.539025 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.539006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token\") pod \"istiod-openshift-gateway-7574459656-b7lxx\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.635281 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.635247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:17.737478 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.737447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:17.814072 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:17.814045 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:17.815726 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:34:17.815699 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe85aad3_915a_4122_8368_92cb68eedd64.slice/crio-7ee9af6ab7e2adad42adb90bd2093691fc0b12cf6e6f312950791c3b8e4f42d2 WatchSource:0}: Error finding container 7ee9af6ab7e2adad42adb90bd2093691fc0b12cf6e6f312950791c3b8e4f42d2: Status 404 returned error can't find the container with id 7ee9af6ab7e2adad42adb90bd2093691fc0b12cf6e6f312950791c3b8e4f42d2 Jul 06 19:34:18.545455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:18.545328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" event={"ID":"be85aad3-915a-4122-8368-92cb68eedd64","Type":"ContainerStarted","Data":"7ee9af6ab7e2adad42adb90bd2093691fc0b12cf6e6f312950791c3b8e4f42d2"} Jul 06 19:34:18.821704 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:18.821608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:20.661273 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:20.661230 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:34:20.661662 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:20.661330 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:34:21.269865 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.269825 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.370082 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.370024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.563237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.563138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" event={"ID":"be85aad3-915a-4122-8368-92cb68eedd64","Type":"ContainerStarted","Data":"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76"} Jul 06 19:34:21.563407 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.563329 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:21.564922 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.564874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" event={"ID":"19a5ba3d-428d-4fd0-82e3-f45a8f81901c","Type":"ContainerStarted","Data":"162a40997c46d868ef5f8b648084f7f77df1839a9cc5f8606a70b3fc07809859"} Jul 06 19:34:21.565222 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.565197 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b7lxx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:34:21.565337 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.565254 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" podUID="be85aad3-915a-4122-8368-92cb68eedd64" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:34:21.593593 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.593536 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" podStartSLOduration=1.750339371 podStartE2EDuration="4.593515791s" totalImagesPullingTime="2.84317642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:21.590789363 +0000 UTC m=+419.244356962" watchObservedRunningTime="2026-07-06 19:34:21.593515791 +0000 UTC m=+419.247083393" Jul 06 19:34:21.611395 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.611346 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58f6467984-49ldb" podStartSLOduration=2.457579989 podStartE2EDuration="6.61133192s" totalImagesPullingTime="4.153751931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:21.609683286 +0000 UTC m=+419.263250897" watchObservedRunningTime="2026-07-06 19:34:21.61133192 +0000 UTC m=+419.264899526" Jul 06 19:34:21.869921 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.869821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.966993 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:21.966957 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:22.156153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:22.156074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:22.577089 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:22.577062 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:27.482546 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.482510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:27.553578 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.553541 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp"] Jul 06 19:34:27.562540 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.562512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.566514 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.566491 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qnj24\"" Jul 06 19:34:27.567665 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.567640 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:34:27.570196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.570171 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr"] Jul 06 19:34:27.573442 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.573417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.576575 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.576557 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9xp5p\"" Jul 06 19:34:27.579335 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.579314 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:34:27.585387 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.585364 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp"] Jul 06 19:34:27.595037 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.595004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hx5\" (UniqueName: \"kubernetes.io/projected/cec36c91-f612-408f-8513-18472a8348b7-kube-api-access-p8hx5\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.595173 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.595047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n772z\" (UniqueName: \"kubernetes.io/projected/1d278012-f523-4671-8938-ee734022253d-kube-api-access-n772z\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.595173 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.595101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.595173 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.595165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec36c91-f612-408f-8513-18472a8348b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.596788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.596764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr"] Jul 06 19:34:27.601482 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.601458 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:27.695123 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.695095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:27.698175 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.698155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec36c91-f612-408f-8513-18472a8348b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.698278 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.698216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hx5\" (UniqueName: \"kubernetes.io/projected/cec36c91-f612-408f-8513-18472a8348b7-kube-api-access-p8hx5\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.698325 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.698307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n772z\" (UniqueName: \"kubernetes.io/projected/1d278012-f523-4671-8938-ee734022253d-kube-api-access-n772z\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.698370 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.698334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.698441 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:34:27.698428 2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 19:34:27.698506 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:34:27.698478 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert podName:1d278012-f523-4671-8938-ee734022253d nodeName:}" failed. No retries permitted until 2026-07-06 19:34:28.198464375 +0000 UTC m=+425.852031952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert") pod "kserve-controller-manager-d6f4b65b8-tw2sp" (UID: "1d278012-f523-4671-8938-ee734022253d") : secret "kserve-webhook-server-cert" not found Jul 06 19:34:27.700534 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.700512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cec36c91-f612-408f-8513-18472a8348b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.712240 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.712216 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n772z\" (UniqueName: \"kubernetes.io/projected/1d278012-f523-4671-8938-ee734022253d-kube-api-access-n772z\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:27.712946 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.712928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hx5\" (UniqueName: \"kubernetes.io/projected/cec36c91-f612-408f-8513-18472a8348b7-kube-api-access-p8hx5\") pod \"llmisvc-controller-manager-777c6bd7d5-zqvrr\" (UID: \"cec36c91-f612-408f-8513-18472a8348b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:27.796082 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.796014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:27.882617 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:27.882589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:28.029405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.029373 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr"] Jul 06 19:34:28.034264 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:34:28.034237 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcec36c91_f612_408f_8513_18472a8348b7.slice/crio-f201604dd150ca97e5d12207839cd013578daa305ad1942b1fda2b1f6f046311 WatchSource:0}: Error finding container f201604dd150ca97e5d12207839cd013578daa305ad1942b1fda2b1f6f046311: Status 404 returned error can't find the container with id f201604dd150ca97e5d12207839cd013578daa305ad1942b1fda2b1f6f046311 Jul 06 19:34:28.203401 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.203331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:28.205606 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.205579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d278012-f523-4671-8938-ee734022253d-cert\") pod \"kserve-controller-manager-d6f4b65b8-tw2sp\" (UID: \"1d278012-f523-4671-8938-ee734022253d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:28.474354 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.474330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:28.603286 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.603258 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp"] Jul 06 19:34:28.603624 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.603576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" event={"ID":"cec36c91-f612-408f-8513-18472a8348b7","Type":"ContainerStarted","Data":"f201604dd150ca97e5d12207839cd013578daa305ad1942b1fda2b1f6f046311"} Jul 06 19:34:28.604679 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:34:28.604655 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d278012_f523_4671_8938_ee734022253d.slice/crio-5d9fe16c70db5646f46541bdf3c4286129dc6781e8d08ef2cbfa954dede80e59 WatchSource:0}: Error finding container 5d9fe16c70db5646f46541bdf3c4286129dc6781e8d08ef2cbfa954dede80e59: Status 404 returned error can't find the container with id 5d9fe16c70db5646f46541bdf3c4286129dc6781e8d08ef2cbfa954dede80e59 Jul 06 19:34:28.855352 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.855288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:28.983541 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:28.983512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:29.612012 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:29.611938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" event={"ID":"1d278012-f523-4671-8938-ee734022253d","Type":"ContainerStarted","Data":"5d9fe16c70db5646f46541bdf3c4286129dc6781e8d08ef2cbfa954dede80e59"} Jul 06 19:34:29.733837 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:29.733800 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:29.734078 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:29.734051 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" podUID="be85aad3-915a-4122-8368-92cb68eedd64" containerName="discovery" containerID="cri-o://300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76" gracePeriod=30 Jul 06 19:34:30.007184 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.007068 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:30.118978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.118936 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119002 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119049 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119095 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdgs2\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119122 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119326 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119326 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119187 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig\") pod \"be85aad3-915a-4122-8368-92cb68eedd64\" (UID: \"be85aad3-915a-4122-8368-92cb68eedd64\") " Jul 06 19:34:30.119326 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119270 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:34:30.119489 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.119386 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-ca-configmap\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.121722 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.121679 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2" (OuterVolumeSpecName: "kube-api-access-qdgs2") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "kube-api-access-qdgs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:30.121857 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.121772 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token" (OuterVolumeSpecName: "istio-token") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:30.121943 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.121888 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts" (OuterVolumeSpecName: "cacerts") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:34:30.122072 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.122039 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:34:30.122376 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.122348 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:34:30.122967 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.122943 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs" (OuterVolumeSpecName: "local-certs") pod "be85aad3-915a-4122-8368-92cb68eedd64" (UID: "be85aad3-915a-4122-8368-92cb68eedd64"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:30.220061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220029 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85aad3-915a-4122-8368-92cb68eedd64-local-certs\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.220061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220061 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgs2\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-kube-api-access-qdgs2\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.220252 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220071 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-cacerts\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.220252 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220079 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-csr-dns-cert\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.220252 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220088 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be85aad3-915a-4122-8368-92cb68eedd64-istio-kubeconfig\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.220252 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.220096 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85aad3-915a-4122-8368-92cb68eedd64-istio-token\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:34:30.619451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.619360 2577 generic.go:356] "Generic (PLEG): container finished" podID="be85aad3-915a-4122-8368-92cb68eedd64" containerID="300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76" exitCode=0 Jul 06 19:34:30.619451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.619422 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" Jul 06 19:34:30.619451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.619430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" event={"ID":"be85aad3-915a-4122-8368-92cb68eedd64","Type":"ContainerDied","Data":"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76"} Jul 06 19:34:30.620035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.619490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx" event={"ID":"be85aad3-915a-4122-8368-92cb68eedd64","Type":"ContainerDied","Data":"7ee9af6ab7e2adad42adb90bd2093691fc0b12cf6e6f312950791c3b8e4f42d2"} Jul 06 19:34:30.620035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.619521 2577 scope.go:117] "RemoveContainer" containerID="300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76" Jul 06 19:34:30.648991 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.648956 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:30.652840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.652817 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b7lxx"] Jul 06 19:34:30.811294 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.811252 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be85aad3-915a-4122-8368-92cb68eedd64" path="/var/lib/kubelet/pods/be85aad3-915a-4122-8368-92cb68eedd64/volumes" Jul 06 19:34:30.856041 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.855983 2577 scope.go:117] "RemoveContainer" containerID="300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76" Jul 06 19:34:30.856327 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:34:30.856299 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76\": container with ID starting with 300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76 not found: ID does not exist" containerID="300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76" Jul 06 19:34:30.856376 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:30.856339 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76"} err="failed to get container status \"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76\": rpc error: code = NotFound desc = could not find container \"300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76\": container with ID starting with 300212703ef96ec884d61ea0ce6d847d1c6eb8098e2d526c32ec025944379e76 not found: ID does not exist" Jul 06 19:34:31.625732 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.625695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" event={"ID":"cec36c91-f612-408f-8513-18472a8348b7","Type":"ContainerStarted","Data":"ea4cabde6697aa02b8ef0393445b5320c0fcdec61f71bf57d58660ebe00d0f5e"} Jul 06 19:34:31.626176 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.625789 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:34:31.627405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.627382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" event={"ID":"1d278012-f523-4671-8938-ee734022253d","Type":"ContainerStarted","Data":"9c11c4c8de663fd0841e676936ec750d1b8cd9d5cbfd46abdc475d6fa0cf119b"} Jul 06 19:34:31.627525 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.627424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:34:31.649675 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.649625 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" podStartSLOduration=1.230555824 podStartE2EDuration="4.649610396s" totalImagesPullingTime="3.419054572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:31.648881991 +0000 UTC m=+429.302449613" watchObservedRunningTime="2026-07-06 19:34:31.649610396 +0000 UTC m=+429.303178004" Jul 06 19:34:31.669584 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:31.669542 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" podStartSLOduration=1.824571754 podStartE2EDuration="4.669526934s" totalImagesPullingTime="2.84495518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:31.668229421 +0000 UTC m=+429.321797020" watchObservedRunningTime="2026-07-06 19:34:31.669526934 +0000 UTC m=+429.323094532" Jul 06 19:34:48.627060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:48.626972 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:50.587684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:50.587656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:50.674861 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:50.674833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:55.053165 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:55.053131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:55.113571 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:34:55.113539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:02.634572 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:02.634542 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zqvrr" Jul 06 19:35:02.637643 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:02.637621 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tw2sp" Jul 06 19:35:16.009629 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.009584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.111079 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.111047 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:16.111341 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.111329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be85aad3-915a-4122-8368-92cb68eedd64" containerName="discovery" Jul 06 19:35:16.111388 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.111344 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be85aad3-915a-4122-8368-92cb68eedd64" containerName="discovery" Jul 06 19:35:16.111423 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.111405 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be85aad3-915a-4122-8368-92cb68eedd64" containerName="discovery" Jul 06 19:35:16.112909 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.112852 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.113339 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.113325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:16.116173 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.116154 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:35:16.116280 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.116214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pdljh\"" Jul 06 19:35:16.116382 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.116367 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:35:16.127475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.127446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:16.184415 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.184386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.189498 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.189481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pl8z\" (UniqueName: \"kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z\") pod \"kuadrant-operator-catalog-vzzj5\" (UID: \"de80b94c-328d-4d84-8d8e-6dd022baac5e\") " pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:16.290457 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.290386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pl8z\" (UniqueName: \"kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z\") pod \"kuadrant-operator-catalog-vzzj5\" (UID: \"de80b94c-328d-4d84-8d8e-6dd022baac5e\") " pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:16.299118 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.299100 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pl8z\" (UniqueName: \"kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z\") pod \"kuadrant-operator-catalog-vzzj5\" (UID: \"de80b94c-328d-4d84-8d8e-6dd022baac5e\") " pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:16.424114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.424082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:16.472634 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.472600 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:16.553004 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.552973 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:16.555551 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:16.555522 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde80b94c_328d_4d84_8d8e_6dd022baac5e.slice/crio-0e9a41deea7e8217f178dc38e6e4269cc633557f20501c42ee5c6ee053478e38 WatchSource:0}: Error finding container 0e9a41deea7e8217f178dc38e6e4269cc633557f20501c42ee5c6ee053478e38: Status 404 returned error can't find the container with id 0e9a41deea7e8217f178dc38e6e4269cc633557f20501c42ee5c6ee053478e38 Jul 06 19:35:16.681990 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.681958 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjpbx"] Jul 06 19:35:16.684803 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.684784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:16.693747 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.693727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhkd\" (UniqueName: \"kubernetes.io/projected/25a32fa0-eb79-4a67-9574-d89640ddca9c-kube-api-access-6hhkd\") pod \"kuadrant-operator-catalog-xjpbx\" (UID: \"25a32fa0-eb79-4a67-9574-d89640ddca9c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:16.694053 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.694032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjpbx"] Jul 06 19:35:16.794948 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.794916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhkd\" (UniqueName: \"kubernetes.io/projected/25a32fa0-eb79-4a67-9574-d89640ddca9c-kube-api-access-6hhkd\") pod \"kuadrant-operator-catalog-xjpbx\" (UID: \"25a32fa0-eb79-4a67-9574-d89640ddca9c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:16.808008 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.807945 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhkd\" (UniqueName: \"kubernetes.io/projected/25a32fa0-eb79-4a67-9574-d89640ddca9c-kube-api-access-6hhkd\") pod \"kuadrant-operator-catalog-xjpbx\" (UID: \"25a32fa0-eb79-4a67-9574-d89640ddca9c\") " pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:16.828698 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.828675 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" event={"ID":"de80b94c-328d-4d84-8d8e-6dd022baac5e","Type":"ContainerStarted","Data":"0e9a41deea7e8217f178dc38e6e4269cc633557f20501c42ee5c6ee053478e38"} Jul 06 19:35:16.908524 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.908486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.995072 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:16.995049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:17.130220 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:17.130195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xjpbx"] Jul 06 19:35:17.132021 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:17.132000 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a32fa0_eb79_4a67_9574_d89640ddca9c.slice/crio-2443e48752ec6f4dcfd35fef79847a347bdab5f23c703480da32b071c9aaf4f2 WatchSource:0}: Error finding container 2443e48752ec6f4dcfd35fef79847a347bdab5f23c703480da32b071c9aaf4f2: Status 404 returned error can't find the container with id 2443e48752ec6f4dcfd35fef79847a347bdab5f23c703480da32b071c9aaf4f2 Jul 06 19:35:17.140706 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:17.140680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:17.836248 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:17.836210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" event={"ID":"25a32fa0-eb79-4a67-9574-d89640ddca9c","Type":"ContainerStarted","Data":"2443e48752ec6f4dcfd35fef79847a347bdab5f23c703480da32b071c9aaf4f2"} Jul 06 19:35:18.843424 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:18.843353 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" event={"ID":"de80b94c-328d-4d84-8d8e-6dd022baac5e","Type":"ContainerStarted","Data":"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3"} Jul 06 19:35:18.843829 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:18.843431 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" podUID="de80b94c-328d-4d84-8d8e-6dd022baac5e" containerName="registry-server" containerID="cri-o://a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3" gracePeriod=2 Jul 06 19:35:18.845563 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:18.845534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" event={"ID":"25a32fa0-eb79-4a67-9574-d89640ddca9c","Type":"ContainerStarted","Data":"bbe4dcd3dc433257f4e54e0ef6a32274f8255c6e5a50edf004b6ce09ebef3c3e"} Jul 06 19:35:18.862641 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:18.862592 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" podStartSLOduration=0.714614492 podStartE2EDuration="2.862576678s" totalImagesPullingTime="2.147962186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:18.860047367 +0000 UTC m=+476.513614967" watchObservedRunningTime="2026-07-06 19:35:18.862576678 +0000 UTC m=+476.516144276" Jul 06 19:35:18.875276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:18.875228 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" podStartSLOduration=1.306899884 podStartE2EDuration="2.875209143s" totalImagesPullingTime="1.568309259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:18.875189398 +0000 UTC m=+476.528756997" watchObservedRunningTime="2026-07-06 19:35:18.875209143 +0000 UTC m=+476.528776741" Jul 06 19:35:19.081088 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.081063 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:19.115503 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.115477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pl8z\" (UniqueName: \"kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z\") pod \"de80b94c-328d-4d84-8d8e-6dd022baac5e\" (UID: \"de80b94c-328d-4d84-8d8e-6dd022baac5e\") " Jul 06 19:35:19.117442 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.117413 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z" (OuterVolumeSpecName: "kube-api-access-9pl8z") pod "de80b94c-328d-4d84-8d8e-6dd022baac5e" (UID: "de80b94c-328d-4d84-8d8e-6dd022baac5e"). InnerVolumeSpecName "kube-api-access-9pl8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:19.215960 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.215940 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pl8z\" (UniqueName: \"kubernetes.io/projected/de80b94c-328d-4d84-8d8e-6dd022baac5e-kube-api-access-9pl8z\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:19.851026 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.850989 2577 generic.go:356] "Generic (PLEG): container finished" podID="de80b94c-328d-4d84-8d8e-6dd022baac5e" containerID="a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3" exitCode=0 Jul 06 19:35:19.851502 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.851031 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" Jul 06 19:35:19.851502 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.851075 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" event={"ID":"de80b94c-328d-4d84-8d8e-6dd022baac5e","Type":"ContainerDied","Data":"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3"} Jul 06 19:35:19.851502 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.851116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vzzj5" event={"ID":"de80b94c-328d-4d84-8d8e-6dd022baac5e","Type":"ContainerDied","Data":"0e9a41deea7e8217f178dc38e6e4269cc633557f20501c42ee5c6ee053478e38"} Jul 06 19:35:19.851502 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.851141 2577 scope.go:117] "RemoveContainer" containerID="a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3" Jul 06 19:35:19.860047 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.860029 2577 scope.go:117] "RemoveContainer" containerID="a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3" Jul 06 19:35:19.860291 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:35:19.860273 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3\": container with ID starting with a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3 not found: ID does not exist" containerID="a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3" Jul 06 19:35:19.860350 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.860303 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3"} err="failed to get container status \"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3\": rpc error: code = NotFound desc = could not find container \"a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3\": container with ID starting with a660c263e95bb0aa17894bee86289480dff14f639fd6747ac44891dafdd65ec3 not found: ID does not exist" Jul 06 19:35:19.872451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.872427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:19.879491 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:19.879472 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vzzj5"] Jul 06 19:35:20.813412 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:20.813376 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de80b94c-328d-4d84-8d8e-6dd022baac5e" path="/var/lib/kubelet/pods/de80b94c-328d-4d84-8d8e-6dd022baac5e/volumes" Jul 06 19:35:26.995998 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:26.995968 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:26.996362 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:26.996040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:27.018195 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:27.018163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:27.911812 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:27.911786 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xjpbx" Jul 06 19:35:31.725237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.725202 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x"] Jul 06 19:35:31.725660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.725497 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de80b94c-328d-4d84-8d8e-6dd022baac5e" containerName="registry-server" Jul 06 19:35:31.725660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.725509 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="de80b94c-328d-4d84-8d8e-6dd022baac5e" containerName="registry-server" Jul 06 19:35:31.725660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.725567 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="de80b94c-328d-4d84-8d8e-6dd022baac5e" containerName="registry-server" Jul 06 19:35:31.730086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.730069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.732748 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.732729 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bvtdd\"" Jul 06 19:35:31.738170 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.738149 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x"] Jul 06 19:35:31.799871 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.799845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.800002 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.799879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.800002 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.799919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4jr\" (UniqueName: \"kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.900954 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.900919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.901134 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.900974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.901134 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.901008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4jr\" (UniqueName: \"kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.901270 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.901250 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.901329 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.901300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:31.910285 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:31.910262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4jr\" (UniqueName: \"kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:32.039245 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.039182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:32.126483 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.126450 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt"] Jul 06 19:35:32.132277 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.132252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.143786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.143761 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt"] Jul 06 19:35:32.173871 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.173840 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x"] Jul 06 19:35:32.177066 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:32.177043 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e29a6c8_cb58_4309_99f0_e27993b54516.slice/crio-ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5 WatchSource:0}: Error finding container ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5: Status 404 returned error can't find the container with id ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5 Jul 06 19:35:32.204157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.204136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.204234 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.204173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.204234 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.204200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7sdw\" (UniqueName: \"kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.305604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.305546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.305604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.305588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7sdw\" (UniqueName: \"kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.305771 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.305632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.305928 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.305884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.305994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.305936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.314559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.314541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7sdw\" (UniqueName: \"kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.443313 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.443284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:32.536136 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.534752 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7"] Jul 06 19:35:32.543455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.543424 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.548513 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.548488 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7"] Jul 06 19:35:32.584401 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.584380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt"] Jul 06 19:35:32.586949 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:32.586923 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f5a4e4_6f1b_4497_8af3_cd8dfab1a8bd.slice/crio-4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31 WatchSource:0}: Error finding container 4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31: Status 404 returned error can't find the container with id 4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31 Jul 06 19:35:32.608061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.608039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.608142 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.608087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.608181 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.608133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7g8g\" (UniqueName: \"kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.708867 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.708833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.708983 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.708895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.708983 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.708949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7g8g\" (UniqueName: \"kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.709272 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.709252 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.709314 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.709266 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.719422 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.719398 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7g8g\" (UniqueName: \"kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.855159 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.855088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:32.916875 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.916848 2577 generic.go:356] "Generic (PLEG): container finished" podID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerID="65ba610abd1f1d76fa42b872095a6ad3d51cf9a932c034f4bfa8c554fc63135f" exitCode=0 Jul 06 19:35:32.917079 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.916939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" event={"ID":"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd","Type":"ContainerDied","Data":"65ba610abd1f1d76fa42b872095a6ad3d51cf9a932c034f4bfa8c554fc63135f"} Jul 06 19:35:32.917079 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.917003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" event={"ID":"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd","Type":"ContainerStarted","Data":"4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31"} Jul 06 19:35:32.918965 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.918933 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerID="4d800ff03875dcb4111f96ad381a51cf1a2440d39fcb7c57c5a6990983dded73" exitCode=0 Jul 06 19:35:32.919339 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.918994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" event={"ID":"5e29a6c8-cb58-4309-99f0-e27993b54516","Type":"ContainerDied","Data":"4d800ff03875dcb4111f96ad381a51cf1a2440d39fcb7c57c5a6990983dded73"} Jul 06 19:35:32.919339 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.919030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" event={"ID":"5e29a6c8-cb58-4309-99f0-e27993b54516","Type":"ContainerStarted","Data":"ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5"} Jul 06 19:35:32.926779 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.926758 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw"] Jul 06 19:35:32.931885 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.931866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:32.940159 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.940032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw"] Jul 06 19:35:32.994013 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:32.993989 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7"] Jul 06 19:35:32.996210 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:32.996185 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3d529c_a24b_4c0e_9dc0_f4e75df1011c.slice/crio-a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629 WatchSource:0}: Error finding container a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629: Status 404 returned error can't find the container with id a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629 Jul 06 19:35:33.012914 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.012877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.013012 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.012961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.013081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.013041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfg84\" (UniqueName: \"kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.114079 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.114025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfg84\" (UniqueName: \"kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.114174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.114082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.114174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.114112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.114399 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.114385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.114438 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.114424 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.124538 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.124517 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfg84\" (UniqueName: \"kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.242400 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.242372 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:33.396015 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.395987 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw"] Jul 06 19:35:33.397274 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:33.397251 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67c54c6_ecc3_4458_b669_af3ed1bd3d50.slice/crio-4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98 WatchSource:0}: Error finding container 4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98: Status 404 returned error can't find the container with id 4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98 Jul 06 19:35:33.925704 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.925676 2577 generic.go:356] "Generic (PLEG): container finished" podID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerID="8370c43a64a5314a6383083ce8dd02e3d03603a2eba342f8182ddf874435c126" exitCode=0 Jul 06 19:35:33.926064 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.925758 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" event={"ID":"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd","Type":"ContainerDied","Data":"8370c43a64a5314a6383083ce8dd02e3d03603a2eba342f8182ddf874435c126"} Jul 06 19:35:33.927556 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.927454 2577 generic.go:356] "Generic (PLEG): container finished" podID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerID="8ab112a15a1b3e1d52d9f9d8e755a801f811a80ad04ec6fde7fb85baca8421d3" exitCode=0 Jul 06 19:35:33.927556 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.927516 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" event={"ID":"a67c54c6-ecc3-4458-b669-af3ed1bd3d50","Type":"ContainerDied","Data":"8ab112a15a1b3e1d52d9f9d8e755a801f811a80ad04ec6fde7fb85baca8421d3"} Jul 06 19:35:33.927668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.927575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" event={"ID":"a67c54c6-ecc3-4458-b669-af3ed1bd3d50","Type":"ContainerStarted","Data":"4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98"} Jul 06 19:35:33.929188 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.929168 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerID="6891598f0dc97d81c982914622e3554507ccc17b48ce6f1a99cfb8ebc25927bc" exitCode=0 Jul 06 19:35:33.929301 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.929258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" event={"ID":"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c","Type":"ContainerDied","Data":"6891598f0dc97d81c982914622e3554507ccc17b48ce6f1a99cfb8ebc25927bc"} Jul 06 19:35:33.929368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.929307 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" event={"ID":"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c","Type":"ContainerStarted","Data":"a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629"} Jul 06 19:35:33.931256 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.931117 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerID="3495ad93e303791ef696eac44af2d20977302424388052ed3ad55214869210de" exitCode=0 Jul 06 19:35:33.931256 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:33.931195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" event={"ID":"5e29a6c8-cb58-4309-99f0-e27993b54516","Type":"ContainerDied","Data":"3495ad93e303791ef696eac44af2d20977302424388052ed3ad55214869210de"} Jul 06 19:35:34.938062 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.937989 2577 generic.go:356] "Generic (PLEG): container finished" podID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerID="ede8fcb26a17d93549a1e99753a5984fe5d110a7de8c36d7605125abc4451573" exitCode=0 Jul 06 19:35:34.938062 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.938024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" event={"ID":"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd","Type":"ContainerDied","Data":"ede8fcb26a17d93549a1e99753a5984fe5d110a7de8c36d7605125abc4451573"} Jul 06 19:35:34.939772 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.939749 2577 generic.go:356] "Generic (PLEG): container finished" podID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerID="5fdc7ebcef5b7c01026e2d6179107920c142069877cc9accd48232cbb1bf2cda" exitCode=0 Jul 06 19:35:34.939885 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.939811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" event={"ID":"a67c54c6-ecc3-4458-b669-af3ed1bd3d50","Type":"ContainerDied","Data":"5fdc7ebcef5b7c01026e2d6179107920c142069877cc9accd48232cbb1bf2cda"} Jul 06 19:35:34.941601 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.941576 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerID="b4bdfb1c7bb1e9317b6eda19fd8ba1f059ee655c3308b5b9d92503e631811743" exitCode=0 Jul 06 19:35:34.941694 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.941656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" event={"ID":"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c","Type":"ContainerDied","Data":"b4bdfb1c7bb1e9317b6eda19fd8ba1f059ee655c3308b5b9d92503e631811743"} Jul 06 19:35:34.943883 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.943863 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerID="becacd6419c17a3eed46d7040b5e59fc5f87b26cf1a330b8c98df4c56d4385d5" exitCode=0 Jul 06 19:35:34.944002 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:34.943940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" event={"ID":"5e29a6c8-cb58-4309-99f0-e27993b54516","Type":"ContainerDied","Data":"becacd6419c17a3eed46d7040b5e59fc5f87b26cf1a330b8c98df4c56d4385d5"} Jul 06 19:35:35.951247 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:35.951213 2577 generic.go:356] "Generic (PLEG): container finished" podID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerID="604c7474b308dc6a954a24afb0cca61148bf709bdc5becf7a38093ae0b054016" exitCode=0 Jul 06 19:35:35.951649 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:35.951290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" event={"ID":"a67c54c6-ecc3-4458-b669-af3ed1bd3d50","Type":"ContainerDied","Data":"604c7474b308dc6a954a24afb0cca61148bf709bdc5becf7a38093ae0b054016"} Jul 06 19:35:35.953237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:35.953216 2577 generic.go:356] "Generic (PLEG): container finished" podID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerID="c7aab7793d79fd67880d244eb057ad0b08c17d0a30349eb622be224517b6dede" exitCode=0 Jul 06 19:35:35.953359 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:35.953296 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" event={"ID":"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c","Type":"ContainerDied","Data":"c7aab7793d79fd67880d244eb057ad0b08c17d0a30349eb622be224517b6dede"} Jul 06 19:35:36.106861 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.106840 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:36.109537 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.109516 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:36.239737 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util\") pod \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " Jul 06 19:35:36.239872 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239746 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7sdw\" (UniqueName: \"kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw\") pod \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " Jul 06 19:35:36.239872 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh4jr\" (UniqueName: \"kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr\") pod \"5e29a6c8-cb58-4309-99f0-e27993b54516\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " Jul 06 19:35:36.239872 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239839 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util\") pod \"5e29a6c8-cb58-4309-99f0-e27993b54516\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " Jul 06 19:35:36.239872 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239856 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle\") pod \"5e29a6c8-cb58-4309-99f0-e27993b54516\" (UID: \"5e29a6c8-cb58-4309-99f0-e27993b54516\") " Jul 06 19:35:36.240116 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.239878 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle\") pod \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\" (UID: \"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd\") " Jul 06 19:35:36.240461 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.240419 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle" (OuterVolumeSpecName: "bundle") pod "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" (UID: "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:36.240461 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.240425 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle" (OuterVolumeSpecName: "bundle") pod "5e29a6c8-cb58-4309-99f0-e27993b54516" (UID: "5e29a6c8-cb58-4309-99f0-e27993b54516"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:36.241997 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.241966 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw" (OuterVolumeSpecName: "kube-api-access-b7sdw") pod "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" (UID: "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd"). InnerVolumeSpecName "kube-api-access-b7sdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:36.242346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.242321 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr" (OuterVolumeSpecName: "kube-api-access-lh4jr") pod "5e29a6c8-cb58-4309-99f0-e27993b54516" (UID: "5e29a6c8-cb58-4309-99f0-e27993b54516"). InnerVolumeSpecName "kube-api-access-lh4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:36.245662 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.245640 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util" (OuterVolumeSpecName: "util") pod "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" (UID: "20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:36.246285 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.246265 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util" (OuterVolumeSpecName: "util") pod "5e29a6c8-cb58-4309-99f0-e27993b54516" (UID: "5e29a6c8-cb58-4309-99f0-e27993b54516"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:36.340482 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340459 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4jr\" (UniqueName: \"kubernetes.io/projected/5e29a6c8-cb58-4309-99f0-e27993b54516-kube-api-access-lh4jr\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.340482 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340481 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.340611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340491 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e29a6c8-cb58-4309-99f0-e27993b54516-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.340611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340499 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.340611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340508 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.340611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.340515 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7sdw\" (UniqueName: \"kubernetes.io/projected/20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd-kube-api-access-b7sdw\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:36.959573 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.959543 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" Jul 06 19:35:36.959924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.959570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt" event={"ID":"20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd","Type":"ContainerDied","Data":"4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31"} Jul 06 19:35:36.959924 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.959614 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff7f85f28190ef4a6eb6bedfb51ff27adc0d3a359fb332580cacfad6dfa4d31" Jul 06 19:35:36.961660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.961638 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" Jul 06 19:35:36.961660 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.961646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x" event={"ID":"5e29a6c8-cb58-4309-99f0-e27993b54516","Type":"ContainerDied","Data":"ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5"} Jul 06 19:35:36.961824 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:36.961689 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff4e33dc2b70b161a6e66f447dc417e75472431751e659ae6b92d5e22930f0b5" Jul 06 19:35:37.086019 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.085994 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:37.122445 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.122424 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:37.148927 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.148880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfg84\" (UniqueName: \"kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84\") pod \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " Jul 06 19:35:37.149060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.148973 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle\") pod \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " Jul 06 19:35:37.149060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.149009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util\") pod \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\" (UID: \"a67c54c6-ecc3-4458-b669-af3ed1bd3d50\") " Jul 06 19:35:37.150372 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.150346 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle" (OuterVolumeSpecName: "bundle") pod "a67c54c6-ecc3-4458-b669-af3ed1bd3d50" (UID: "a67c54c6-ecc3-4458-b669-af3ed1bd3d50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:37.151465 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.151435 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84" (OuterVolumeSpecName: "kube-api-access-sfg84") pod "a67c54c6-ecc3-4458-b669-af3ed1bd3d50" (UID: "a67c54c6-ecc3-4458-b669-af3ed1bd3d50"). InnerVolumeSpecName "kube-api-access-sfg84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:37.155364 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.155340 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util" (OuterVolumeSpecName: "util") pod "a67c54c6-ecc3-4458-b669-af3ed1bd3d50" (UID: "a67c54c6-ecc3-4458-b669-af3ed1bd3d50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:37.249915 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.249870 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7g8g\" (UniqueName: \"kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g\") pod \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " Jul 06 19:35:37.250041 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.249966 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle\") pod \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " Jul 06 19:35:37.250041 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.249992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util\") pod \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\" (UID: \"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c\") " Jul 06 19:35:37.250141 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.250128 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfg84\" (UniqueName: \"kubernetes.io/projected/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-kube-api-access-sfg84\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.250176 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.250145 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.250176 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.250153 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67c54c6-ecc3-4458-b669-af3ed1bd3d50-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.250386 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.250362 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle" (OuterVolumeSpecName: "bundle") pod "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" (UID: "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:37.251853 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.251829 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g" (OuterVolumeSpecName: "kube-api-access-d7g8g") pod "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" (UID: "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c"). InnerVolumeSpecName "kube-api-access-d7g8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:37.255464 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.255430 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util" (OuterVolumeSpecName: "util") pod "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" (UID: "8a3d529c-a24b-4c0e-9dc0-f4e75df1011c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:37.350987 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.350931 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7g8g\" (UniqueName: \"kubernetes.io/projected/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-kube-api-access-d7g8g\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.350987 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.350952 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-bundle\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.350987 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.350960 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3d529c-a24b-4c0e-9dc0-f4e75df1011c-util\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:35:37.967555 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.967532 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" Jul 06 19:35:37.967991 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.967522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw" event={"ID":"a67c54c6-ecc3-4458-b669-af3ed1bd3d50","Type":"ContainerDied","Data":"4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98"} Jul 06 19:35:37.967991 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.967659 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cde574aec786b6e1fa7724b5c5f8e6a76b69a1a2a201204fcd4bb0e28d94f98" Jul 06 19:35:37.969541 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.969523 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" Jul 06 19:35:37.969658 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.969561 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7" event={"ID":"8a3d529c-a24b-4c0e-9dc0-f4e75df1011c","Type":"ContainerDied","Data":"a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629"} Jul 06 19:35:37.969658 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:37.969596 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a74fe57bfc361fb49748d990afc119e95cbcf8c9a55884e92949d621f4e35629" Jul 06 19:35:39.284090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:39.284030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:39.865892 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:39.865858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:40.189249 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:40.189171 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:42.772871 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.772840 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qqngq"] Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773130 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773141 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773152 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773157 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773165 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773171 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773180 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773187 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773192 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773197 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="util" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773204 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="extract" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773208 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="extract" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773217 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773222 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773228 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773232 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="pull" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773238 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="extract" Jul 06 19:35:42.773233 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773243 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773248 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773253 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773265 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="util" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773270 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="util" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773275 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773280 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773327 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3d529c-a24b-4c0e-9dc0-f4e75df1011c" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773336 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e29a6c8-cb58-4309-99f0-e27993b54516" containerName="extract" Jul 06 19:35:42.773762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.773349 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67c54c6-ecc3-4458-b669-af3ed1bd3d50" containerName="extract" Jul 06 19:35:42.777263 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.777242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:42.779884 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.779869 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-sdvtg\"" Jul 06 19:35:42.788929 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.788889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qqngq"] Jul 06 19:35:42.892117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.892084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7g2f\" (UniqueName: \"kubernetes.io/projected/468797fc-f8ed-4b0b-9110-8db59469ddc6-kube-api-access-t7g2f\") pod \"authorino-operator-75cc8d5cb7-qqngq\" (UID: \"468797fc-f8ed-4b0b-9110-8db59469ddc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:42.993455 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:42.993418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7g2f\" (UniqueName: \"kubernetes.io/projected/468797fc-f8ed-4b0b-9110-8db59469ddc6-kube-api-access-t7g2f\") pod \"authorino-operator-75cc8d5cb7-qqngq\" (UID: \"468797fc-f8ed-4b0b-9110-8db59469ddc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:43.003216 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:43.003193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7g2f\" (UniqueName: \"kubernetes.io/projected/468797fc-f8ed-4b0b-9110-8db59469ddc6-kube-api-access-t7g2f\") pod \"authorino-operator-75cc8d5cb7-qqngq\" (UID: \"468797fc-f8ed-4b0b-9110-8db59469ddc6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:43.086971 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:43.086891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:43.225020 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:43.224995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qqngq"] Jul 06 19:35:43.227724 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:43.227696 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468797fc_f8ed_4b0b_9110_8db59469ddc6.slice/crio-ce24b121860523db4ef36f8654063207853d99ccae6049a4d16409319cb3ef67 WatchSource:0}: Error finding container ce24b121860523db4ef36f8654063207853d99ccae6049a4d16409319cb3ef67: Status 404 returned error can't find the container with id ce24b121860523db4ef36f8654063207853d99ccae6049a4d16409319cb3ef67 Jul 06 19:35:43.999747 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:43.999703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" event={"ID":"468797fc-f8ed-4b0b-9110-8db59469ddc6","Type":"ContainerStarted","Data":"ce24b121860523db4ef36f8654063207853d99ccae6049a4d16409319cb3ef67"} Jul 06 19:35:44.104722 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:44.104651 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.692196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:44.692169 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.797029 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:44.796990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.896074 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:44.896041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.993578 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:44.993545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:45.402556 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:45.402471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:45.501643 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:45.501622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:46.011638 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.011604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" event={"ID":"468797fc-f8ed-4b0b-9110-8db59469ddc6","Type":"ContainerStarted","Data":"0d25008f1c50da17222362766d7f652619453723a854906803da2f3de096132d"} Jul 06 19:35:46.011803 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.011743 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:46.038350 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.038309 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" podStartSLOduration=1.797873226 podStartE2EDuration="4.038296119s" totalImagesPullingTime="2.240422893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:46.03599942 +0000 UTC m=+503.689567031" watchObservedRunningTime="2026-07-06 19:35:46.038296119 +0000 UTC m=+503.691863718" Jul 06 19:35:46.390007 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.389921 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w"] Jul 06 19:35:46.393032 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.393013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:46.397158 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.397138 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:35:46.397250 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.397146 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sftpt\"" Jul 06 19:35:46.415200 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.415179 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w"] Jul 06 19:35:46.524946 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.524921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9fw\" (UniqueName: \"kubernetes.io/projected/25cd4103-60d0-4cb7-9295-5a664accc94f-kube-api-access-5p9fw\") pod \"dns-operator-controller-manager-864dd78c4c-b2m9w\" (UID: \"25cd4103-60d0-4cb7-9295-5a664accc94f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:46.630661 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.626102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9fw\" (UniqueName: \"kubernetes.io/projected/25cd4103-60d0-4cb7-9295-5a664accc94f-kube-api-access-5p9fw\") pod \"dns-operator-controller-manager-864dd78c4c-b2m9w\" (UID: \"25cd4103-60d0-4cb7-9295-5a664accc94f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:46.637820 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.637794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9fw\" (UniqueName: \"kubernetes.io/projected/25cd4103-60d0-4cb7-9295-5a664accc94f-kube-api-access-5p9fw\") pod \"dns-operator-controller-manager-864dd78c4c-b2m9w\" (UID: \"25cd4103-60d0-4cb7-9295-5a664accc94f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:46.702615 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.702554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:46.843000 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:46.842977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w"] Jul 06 19:35:46.845276 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:46.845245 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cd4103_60d0_4cb7_9295_5a664accc94f.slice/crio-a2ff001a1844caa1fdc063e9dca473d3145b4f44c2830c36119b2e7f61fa24ce WatchSource:0}: Error finding container a2ff001a1844caa1fdc063e9dca473d3145b4f44c2830c36119b2e7f61fa24ce: Status 404 returned error can't find the container with id a2ff001a1844caa1fdc063e9dca473d3145b4f44c2830c36119b2e7f61fa24ce Jul 06 19:35:47.018266 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:47.018233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" event={"ID":"25cd4103-60d0-4cb7-9295-5a664accc94f","Type":"ContainerStarted","Data":"a2ff001a1844caa1fdc063e9dca473d3145b4f44c2830c36119b2e7f61fa24ce"} Jul 06 19:35:47.480998 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:47.480943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:47.578421 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:47.578391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:49.028865 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.028831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" event={"ID":"25cd4103-60d0-4cb7-9295-5a664accc94f","Type":"ContainerStarted","Data":"9a06d2987514d8080b67153d33d0bdbfdde3c569f43ca7d94d91e58276a95b23"} Jul 06 19:35:49.029266 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.028938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:35:49.053163 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.053123 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" podStartSLOduration=1.351001767 podStartE2EDuration="3.05311119s" totalImagesPullingTime="1.702109423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:49.050868812 +0000 UTC m=+506.704436411" watchObservedRunningTime="2026-07-06 19:35:49.05311119 +0000 UTC m=+506.706678788" Jul 06 19:35:49.753548 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.753513 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf"] Jul 06 19:35:49.757028 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.757008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:49.760210 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.760192 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6x487\"" Jul 06 19:35:49.768156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.768136 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf"] Jul 06 19:35:49.850179 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.850148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dvs\" (UniqueName: \"kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs\") pod \"limitador-operator-controller-manager-859d7b78bd-smsvf\" (UID: \"74262925-8006-42f0-85e0-56a98747878f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:49.951434 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.951401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dvs\" (UniqueName: \"kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs\") pod \"limitador-operator-controller-manager-859d7b78bd-smsvf\" (UID: \"74262925-8006-42f0-85e0-56a98747878f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:49.963326 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.963288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:49.965354 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:49.965332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dvs\" (UniqueName: \"kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs\") pod \"limitador-operator-controller-manager-859d7b78bd-smsvf\" (UID: \"74262925-8006-42f0-85e0-56a98747878f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:50.067680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:50.067598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:50.117238 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:50.116842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:50.222069 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:50.222039 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf"] Jul 06 19:35:50.224912 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:50.224877 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74262925_8006_42f0_85e0_56a98747878f.slice/crio-7ea6705d57a67210a1b29462d72ccb49d592c2ef204ee827cb2d11fc1fdb7545 WatchSource:0}: Error finding container 7ea6705d57a67210a1b29462d72ccb49d592c2ef204ee827cb2d11fc1fdb7545: Status 404 returned error can't find the container with id 7ea6705d57a67210a1b29462d72ccb49d592c2ef204ee827cb2d11fc1fdb7545 Jul 06 19:35:51.040998 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:51.040965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" event={"ID":"74262925-8006-42f0-85e0-56a98747878f","Type":"ContainerStarted","Data":"7ea6705d57a67210a1b29462d72ccb49d592c2ef204ee827cb2d11fc1fdb7545"} Jul 06 19:35:52.048287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:52.048260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" event={"ID":"74262925-8006-42f0-85e0-56a98747878f","Type":"ContainerStarted","Data":"ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf"} Jul 06 19:35:52.048639 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:52.048339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:35:52.086215 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:52.084855 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" podStartSLOduration=1.342874842 podStartE2EDuration="3.084832828s" totalImagesPullingTime="1.741957986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:52.07925821 +0000 UTC m=+509.732825831" watchObservedRunningTime="2026-07-06 19:35:52.084832828 +0000 UTC m=+509.738400429" Jul 06 19:35:52.784653 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:52.784607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:53.639095 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.639054 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5"] Jul 06 19:35:53.647490 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.647468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.650458 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.650435 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z899f\"" Jul 06 19:35:53.656399 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.656379 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5"] Jul 06 19:35:53.783668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.783639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.783816 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.783676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2cs\" (UniqueName: \"kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.884681 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.884650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.884786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.884695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2cs\" (UniqueName: \"kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.885013 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.884996 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.899960 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.899891 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2cs\" (UniqueName: \"kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:53.958416 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:53.958393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:35:54.098490 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:54.098466 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5"] Jul 06 19:35:54.100339 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:35:54.100319 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1ed847_5752_4e34_8bec_06259b688e79.slice/crio-b5453c74799ce1dc0b5d1b69df6852fb6a12e9a6c6826f3e82c504573bdafc9e WatchSource:0}: Error finding container b5453c74799ce1dc0b5d1b69df6852fb6a12e9a6c6826f3e82c504573bdafc9e: Status 404 returned error can't find the container with id b5453c74799ce1dc0b5d1b69df6852fb6a12e9a6c6826f3e82c504573bdafc9e Jul 06 19:35:54.888205 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:54.888172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:54.983279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:54.983246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:55.064951 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:55.064920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" event={"ID":"1a1ed847-5752-4e34-8bec-06259b688e79","Type":"ContainerStarted","Data":"b5453c74799ce1dc0b5d1b69df6852fb6a12e9a6c6826f3e82c504573bdafc9e"} Jul 06 19:35:55.241673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:55.241645 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:55.340221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:55.340195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:55.437267 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:55.437236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:57.020888 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:57.020864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qqngq" Jul 06 19:35:58.883165 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:58.883094 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:58.979894 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:35:58.979861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:00.035718 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:00.035690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2m9w" Jul 06 19:36:00.091811 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:00.091764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" event={"ID":"1a1ed847-5752-4e34-8bec-06259b688e79","Type":"ContainerStarted","Data":"69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e"} Jul 06 19:36:00.092055 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:00.092003 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:00.117442 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:00.117387 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" podStartSLOduration=2.078679049 podStartE2EDuration="7.117371103s" totalImagesPullingTime="5.038692054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:00.116076765 +0000 UTC m=+517.769644365" watchObservedRunningTime="2026-07-06 19:36:00.117371103 +0000 UTC m=+517.770938701" Jul 06 19:36:01.706426 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.706393 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q"] Jul 06 19:36:01.709796 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.709779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.713201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.713173 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:36:01.713448 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.713416 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:36:01.713568 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.713462 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bvtdd\"" Jul 06 19:36:01.722656 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.722635 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q"] Jul 06 19:36:01.737586 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.737559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:01.848366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.848337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6x24\" (UniqueName: \"kubernetes.io/projected/9cca4385-a358-42b3-aac7-0a538a7724ae-kube-api-access-t6x24\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.848518 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.848398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca4385-a358-42b3-aac7-0a538a7724ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.848518 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.848481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cca4385-a358-42b3-aac7-0a538a7724ae-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.949292 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.949262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6x24\" (UniqueName: \"kubernetes.io/projected/9cca4385-a358-42b3-aac7-0a538a7724ae-kube-api-access-t6x24\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.949516 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.949494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca4385-a358-42b3-aac7-0a538a7724ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.949608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.949549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cca4385-a358-42b3-aac7-0a538a7724ae-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.950246 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.950224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cca4385-a358-42b3-aac7-0a538a7724ae-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.951910 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.951878 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cca4385-a358-42b3-aac7-0a538a7724ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:01.958275 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:01.958223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6x24\" (UniqueName: \"kubernetes.io/projected/9cca4385-a358-42b3-aac7-0a538a7724ae-kube-api-access-t6x24\") pod \"kuadrant-console-plugin-66b5774985-7wx8q\" (UID: \"9cca4385-a358-42b3-aac7-0a538a7724ae\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:02.015125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:02.015104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:02.019116 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:02.019100 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" Jul 06 19:36:02.150876 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:02.150845 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q"] Jul 06 19:36:02.154097 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:36:02.154064 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cca4385_a358_42b3_aac7_0a538a7724ae.slice/crio-e979108419d8aa526cb053f056f1161ed198edd2f8a4dea7712f260d4d05ed32 WatchSource:0}: Error finding container e979108419d8aa526cb053f056f1161ed198edd2f8a4dea7712f260d4d05ed32: Status 404 returned error can't find the container with id e979108419d8aa526cb053f056f1161ed198edd2f8a4dea7712f260d4d05ed32 Jul 06 19:36:03.054640 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:03.054613 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:03.086349 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:03.086316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:03.109778 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:03.109746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" event={"ID":"9cca4385-a358-42b3-aac7-0a538a7724ae","Type":"ContainerStarted","Data":"e979108419d8aa526cb053f056f1161ed198edd2f8a4dea7712f260d4d05ed32"} Jul 06 19:36:03.184718 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:03.184679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:04.930572 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:04.930460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:06.497366 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:06.497320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:06.597155 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:06.597084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:11.099957 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:11.099924 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:12.411618 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.411585 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb"] Jul 06 19:36:12.425663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.425640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.433818 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.433792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb"] Jul 06 19:36:12.441451 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.441417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbbm\" (UniqueName: \"kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.441587 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.441475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.542476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.542440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbbm\" (UniqueName: \"kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.542476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.542489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.542883 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.542857 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.555878 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.555849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbbm\" (UniqueName: \"kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm\") pod \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.739184 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.739155 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:12.757424 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.757393 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb"] Jul 06 19:36:12.768631 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.768602 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb"] Jul 06 19:36:12.778996 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.778975 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5"] Jul 06 19:36:12.779201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.779174 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" containerName="manager" containerID="cri-o://69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e" gracePeriod=2 Jul 06 19:36:12.783180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.783156 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:12.786874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.786852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:12.796681 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.796657 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5"] Jul 06 19:36:12.804731 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:12.804692 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1a1ed847-5752-4e34-8bec-06259b688e79\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:12.810580 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:12.810552 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:12.812026 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.811954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:12.812026 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.812003 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:12.812452 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.812430 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" containerName="manager" Jul 06 19:36:12.812533 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.812453 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" containerName="manager" Jul 06 19:36:12.812584 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.812553 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" containerName="manager" Jul 06 19:36:12.817084 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.816225 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf"] Jul 06 19:36:12.817084 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.816400 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" podUID="74262925-8006-42f0-85e0-56a98747878f" containerName="manager" containerID="cri-o://ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf" gracePeriod=2 Jul 06 19:36:12.817084 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.816576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:12.828704 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.828636 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf"] Jul 06 19:36:12.831169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.831147 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:12.847749 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.847722 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2"] Jul 06 19:36:12.848203 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.848186 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74262925-8006-42f0-85e0-56a98747878f" containerName="manager" Jul 06 19:36:12.848279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.848206 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74262925-8006-42f0-85e0-56a98747878f" containerName="manager" Jul 06 19:36:12.848331 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.848308 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74262925-8006-42f0-85e0-56a98747878f" containerName="manager" Jul 06 19:36:12.851479 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.851459 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:12.861317 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.861292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2"] Jul 06 19:36:12.865652 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:12.865602 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:12.891753 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:12.891716 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:12.945890 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.945861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9l8d\" (UniqueName: \"kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:12.946003 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.945940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:12.946142 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.946115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7flh\" (UniqueName: \"kubernetes.io/projected/4ef0ccf7-0681-493e-9306-54985188afd6-kube-api-access-j7flh\") pod \"limitador-operator-controller-manager-859d7b78bd-c5dv2\" (UID: \"4ef0ccf7-0681-493e-9306-54985188afd6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:12.946289 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.946265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:12.946364 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:12.946301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxrv\" (UniqueName: \"kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.046917 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.046825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9l8d\" (UniqueName: \"kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:13.046917 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.046884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:13.047114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.046929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7flh\" (UniqueName: \"kubernetes.io/projected/4ef0ccf7-0681-493e-9306-54985188afd6-kube-api-access-j7flh\") pod \"limitador-operator-controller-manager-859d7b78bd-c5dv2\" (UID: \"4ef0ccf7-0681-493e-9306-54985188afd6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:13.047114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.047040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.047114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.047080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxrv\" (UniqueName: \"kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.047511 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.047473 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:13.047511 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.047493 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.060653 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.060630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9l8d\" (UniqueName: \"kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d\") pod \"kuadrant-operator-controller-manager-55467d9f7-tv8kf\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:13.060770 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.060737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxrv\" (UniqueName: \"kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-p9khz\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.060959 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.060938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7flh\" (UniqueName: \"kubernetes.io/projected/4ef0ccf7-0681-493e-9306-54985188afd6-kube-api-access-j7flh\") pod \"limitador-operator-controller-manager-859d7b78bd-c5dv2\" (UID: \"4ef0ccf7-0681-493e-9306-54985188afd6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:13.187245 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.187217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:13.197287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.197259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:13.203346 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:13.203324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:14.053102 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:14.053060 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" podUID="74262925-8006-42f0-85e0-56a98747878f" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 19:36:14.417777 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:14.417697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:14.523156 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:14.523113 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:14.981670 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:14.981635 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:15.084953 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:15.084849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:17.343269 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:17.343232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:17.399510 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:17.399465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.560363 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:22.560320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.658633 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:22.658579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.875949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.875667 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:24.877552 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:36:24.876958 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08d40dd_0f22_4ed4_8c88_ce285cca11c0.slice/crio-65f35351ae7cb241714a32af706f19688f7d5096e963d53a7ba410844b385a66 WatchSource:0}: Error finding container 65f35351ae7cb241714a32af706f19688f7d5096e963d53a7ba410844b385a66: Status 404 returned error can't find the container with id 65f35351ae7cb241714a32af706f19688f7d5096e963d53a7ba410844b385a66 Jul 06 19:36:24.886831 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.886814 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:24.889034 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.888992 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:24.889762 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:24.889730 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:24.892587 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:24.892546 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:24.895183 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:24.895152 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:24.912081 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.912056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2"] Jul 06 19:36:24.912180 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:36:24.912114 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef0ccf7_0681_493e_9306_54985188afd6.slice/crio-59104e198dfcffdee94753bf3643ca6196c09715b7c4906c82eea1d6388e2e73 WatchSource:0}: Error finding container 59104e198dfcffdee94753bf3643ca6196c09715b7c4906c82eea1d6388e2e73: Status 404 returned error can't find the container with id 59104e198dfcffdee94753bf3643ca6196c09715b7c4906c82eea1d6388e2e73 Jul 06 19:36:24.948354 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.948327 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9dvs\" (UniqueName: \"kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs\") pod \"74262925-8006-42f0-85e0-56a98747878f\" (UID: \"74262925-8006-42f0-85e0-56a98747878f\") " Jul 06 19:36:24.948470 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.948395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b2cs\" (UniqueName: \"kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs\") pod \"1a1ed847-5752-4e34-8bec-06259b688e79\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " Jul 06 19:36:24.948535 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.948471 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume\") pod \"1a1ed847-5752-4e34-8bec-06259b688e79\" (UID: \"1a1ed847-5752-4e34-8bec-06259b688e79\") " Jul 06 19:36:24.949204 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.949163 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1a1ed847-5752-4e34-8bec-06259b688e79" (UID: "1a1ed847-5752-4e34-8bec-06259b688e79"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:36:24.950432 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.950410 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs" (OuterVolumeSpecName: "kube-api-access-x9dvs") pod "74262925-8006-42f0-85e0-56a98747878f" (UID: "74262925-8006-42f0-85e0-56a98747878f"). InnerVolumeSpecName "kube-api-access-x9dvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:24.951180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:24.951154 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs" (OuterVolumeSpecName: "kube-api-access-2b2cs") pod "1a1ed847-5752-4e34-8bec-06259b688e79" (UID: "1a1ed847-5752-4e34-8bec-06259b688e79"). InnerVolumeSpecName "kube-api-access-2b2cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:25.049734 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.049709 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a1ed847-5752-4e34-8bec-06259b688e79-extensions-socket-volume\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:25.049734 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.049734 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9dvs\" (UniqueName: \"kubernetes.io/projected/74262925-8006-42f0-85e0-56a98747878f-kube-api-access-x9dvs\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:25.049886 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.049747 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2cs\" (UniqueName: \"kubernetes.io/projected/1a1ed847-5752-4e34-8bec-06259b688e79-kube-api-access-2b2cs\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:25.141758 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.141729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:25.142479 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:36:25.142452 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ed648c_1d43_4a87_ae47_ab6e6829f84a.slice/crio-67b5c2153d6f84bcb43c092a4908d1c64297c9735af49e3b218fb08937e3eeaa WatchSource:0}: Error finding container 67b5c2153d6f84bcb43c092a4908d1c64297c9735af49e3b218fb08937e3eeaa: Status 404 returned error can't find the container with id 67b5c2153d6f84bcb43c092a4908d1c64297c9735af49e3b218fb08937e3eeaa Jul 06 19:36:25.195949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.195924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:25.234827 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.234712 2577 generic.go:356] "Generic (PLEG): container finished" podID="74262925-8006-42f0-85e0-56a98747878f" containerID="ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf" exitCode=0 Jul 06 19:36:25.234955 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.234852 2577 scope.go:117] "RemoveContainer" containerID="ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf" Jul 06 19:36:25.237217 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.235067 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:25.238714 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.238680 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:25.245301 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.243726 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:25.245301 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.244977 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" event={"ID":"f0ed648c-1d43-4a87-ae47-ab6e6829f84a","Type":"ContainerStarted","Data":"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94"} Jul 06 19:36:25.245301 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.245239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" event={"ID":"f0ed648c-1d43-4a87-ae47-ab6e6829f84a","Type":"ContainerStarted","Data":"67b5c2153d6f84bcb43c092a4908d1c64297c9735af49e3b218fb08937e3eeaa"} Jul 06 19:36:25.246267 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.245754 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:25.248405 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.248376 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:25.248733 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.248704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" event={"ID":"9cca4385-a358-42b3-aac7-0a538a7724ae","Type":"ContainerStarted","Data":"bffed59084f14fe917b548cbe0e7a9fb6582e50b60832696c76288a24d2c13f0"} Jul 06 19:36:25.250533 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.250508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" event={"ID":"4ef0ccf7-0681-493e-9306-54985188afd6","Type":"ContainerStarted","Data":"1f474f7068be615eb0202cd7d3b3b7e19b37f065a6b043c8feb43e4528a643e2"} Jul 06 19:36:25.250642 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.250557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" event={"ID":"4ef0ccf7-0681-493e-9306-54985188afd6","Type":"ContainerStarted","Data":"59104e198dfcffdee94753bf3643ca6196c09715b7c4906c82eea1d6388e2e73"} Jul 06 19:36:25.250710 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.250660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:25.251786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.251765 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a1ed847-5752-4e34-8bec-06259b688e79" containerID="69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e" exitCode=0 Jul 06 19:36:25.251877 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.251802 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:25.253949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.253880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" event={"ID":"d08d40dd-0f22-4ed4-8c88-ce285cca11c0","Type":"ContainerStarted","Data":"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437"} Jul 06 19:36:25.254039 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.253955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" event={"ID":"d08d40dd-0f22-4ed4-8c88-ce285cca11c0","Type":"ContainerStarted","Data":"65f35351ae7cb241714a32af706f19688f7d5096e963d53a7ba410844b385a66"} Jul 06 19:36:25.254196 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.254183 2577 scope.go:117] "RemoveContainer" containerID="ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf" Jul 06 19:36:25.254432 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.254415 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf\": container with ID starting with ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf not found: ID does not exist" containerID="ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf" Jul 06 19:36:25.254497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.254437 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf"} err="failed to get container status \"ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf\": rpc error: code = NotFound desc = could not find container \"ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf\": container with ID starting with ca10a281f203619e451dff9b6564b713dd41ddeb6eb71e0aaa8fa3d0d0e3d0bf not found: ID does not exist" Jul 06 19:36:25.254497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.254453 2577 scope.go:117] "RemoveContainer" containerID="69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e" Jul 06 19:36:25.263110 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.263091 2577 scope.go:117] "RemoveContainer" containerID="69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e" Jul 06 19:36:25.263345 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.263326 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e\": container with ID starting with 69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e not found: ID does not exist" containerID="69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e" Jul 06 19:36:25.263389 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.263353 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e"} err="failed to get container status \"69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e\": rpc error: code = NotFound desc = could not find container \"69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e\": container with ID starting with 69c45e89fdd8acee0b6d94854f02a7ae8827130d4d047f1fe7264abdb34a902e not found: ID does not exist" Jul 06 19:36:25.272119 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.272056 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" podStartSLOduration=13.272040839 podStartE2EDuration="13.272040839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:25.2705602 +0000 UTC m=+542.924127798" watchObservedRunningTime="2026-07-06 19:36:25.272040839 +0000 UTC m=+542.925608439" Jul 06 19:36:25.273035 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.273015 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:25.275606 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.275588 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6xbb5\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6xbb5" Jul 06 19:36:25.292912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.292880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:25.299409 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.299376 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" podStartSLOduration=13.299364522 podStartE2EDuration="13.299364522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:25.297190802 +0000 UTC m=+542.950758400" watchObservedRunningTime="2026-07-06 19:36:25.299364522 +0000 UTC m=+542.952932111" Jul 06 19:36:25.323357 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:25.323321 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7wx8q" podStartSLOduration=1.68844325 podStartE2EDuration="24.32331126s" totalImagesPullingTime="22.63486801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:25.322684366 +0000 UTC m=+542.976251962" watchObservedRunningTime="2026-07-06 19:36:25.32331126 +0000 UTC m=+542.976878859" Jul 06 19:36:25.324743 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:25.324722 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smsvf\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="74262925-8006-42f0-85e0-56a98747878f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smsvf" Jul 06 19:36:26.261922 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:26.261873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:26.809315 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:26.809278 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1ed847-5752-4e34-8bec-06259b688e79" path="/var/lib/kubelet/pods/1a1ed847-5752-4e34-8bec-06259b688e79/volumes" Jul 06 19:36:26.809626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:26.809612 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74262925-8006-42f0-85e0-56a98747878f" path="/var/lib/kubelet/pods/74262925-8006-42f0-85e0-56a98747878f/volumes" Jul 06 19:36:27.258101 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:27.258058 2577 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-bmkzb_kuadrant-system_8a2ab755-12f5-4cc8-bd7e-5f5385e4827a_0(0c55aa70ba42584259867b0102f6304f7d73be0faefd416ac177bcdbcf3a01f5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-bmkzb to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 06 19:36:27.258162 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:27.258127 2577 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-bmkzb_kuadrant-system_8a2ab755-12f5-4cc8-bd7e-5f5385e4827a_0(0c55aa70ba42584259867b0102f6304f7d73be0faefd416ac177bcdbcf3a01f5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-bmkzb to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:27.266327 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.266305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:27.269379 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:27.269356 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:27.275052 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.275033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:27.277696 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:27.277674 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:27.366349 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.366324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume\") pod \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " Jul 06 19:36:27.366441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.366426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrbbm\" (UniqueName: \"kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm\") pod \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\" (UID: \"8a2ab755-12f5-4cc8-bd7e-5f5385e4827a\") " Jul 06 19:36:27.366624 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.366597 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" (UID: "8a2ab755-12f5-4cc8-bd7e-5f5385e4827a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:36:27.368323 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.368303 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm" (OuterVolumeSpecName: "kube-api-access-mrbbm") pod "8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" (UID: "8a2ab755-12f5-4cc8-bd7e-5f5385e4827a"). InnerVolumeSpecName "kube-api-access-mrbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:27.467887 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.467864 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbbm\" (UniqueName: \"kubernetes.io/projected/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-kube-api-access-mrbbm\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:27.467887 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:27.467885 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a-extensions-socket-volume\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:28.270737 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:28.270703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:28.273667 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:28.273635 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:28.279317 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:28.279286 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-bmkzb\" is forbidden: User \"system:node:ip-10-0-130-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-143.ec2.internal' and this object" podUID="8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-bmkzb" Jul 06 19:36:28.810201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:28.810170 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2ab755-12f5-4cc8-bd7e-5f5385e4827a" path="/var/lib/kubelet/pods/8a2ab755-12f5-4cc8-bd7e-5f5385e4827a/volumes" Jul 06 19:36:29.107091 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:29.107013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:29.211720 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:29.211692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:36.263510 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:36.263477 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:36.263991 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:36.263545 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" Jul 06 19:36:36.289762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:36.289710 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c5dv2" podStartSLOduration=24.28969705 podStartE2EDuration="24.28969705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:25.352076375 +0000 UTC m=+543.005643976" watchObservedRunningTime="2026-07-06 19:36:36.28969705 +0000 UTC m=+553.943264649" Jul 06 19:36:37.269294 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.269264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:37.323192 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.323160 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:37.323528 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.323491 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" podUID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" containerName="manager" containerID="cri-o://0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94" gracePeriod=10 Jul 06 19:36:37.532247 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.532178 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:36:37.535634 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.535616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.554308 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.554282 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:36:37.578076 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.578056 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:37.638375 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.638351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnxrv\" (UniqueName: \"kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv\") pod \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " Jul 06 19:36:37.638484 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.638457 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume\") pod \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\" (UID: \"f0ed648c-1d43-4a87-ae47-ab6e6829f84a\") " Jul 06 19:36:37.638578 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.638560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fff6h\" (UniqueName: \"kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.638641 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.638625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.638845 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.638815 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0ed648c-1d43-4a87-ae47-ab6e6829f84a" (UID: "f0ed648c-1d43-4a87-ae47-ab6e6829f84a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:36:37.640594 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.640562 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv" (OuterVolumeSpecName: "kube-api-access-gnxrv") pod "f0ed648c-1d43-4a87-ae47-ab6e6829f84a" (UID: "f0ed648c-1d43-4a87-ae47-ab6e6829f84a"). InnerVolumeSpecName "kube-api-access-gnxrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:37.739414 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.739391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.739545 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.739455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fff6h\" (UniqueName: \"kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.739545 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.739487 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxrv\" (UniqueName: \"kubernetes.io/projected/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-kube-api-access-gnxrv\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:37.739545 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.739497 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ed648c-1d43-4a87-ae47-ab6e6829f84a-extensions-socket-volume\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:37.739743 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.739725 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.751351 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.751327 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fff6h\" (UniqueName: \"kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h\") pod \"kuadrant-operator-controller-manager-86878c769f-dfx2r\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.845516 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.845460 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:37.981060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:37.981034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:36:37.983627 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:36:37.983603 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9458a0_518c_4292_bf9f_d6ed6937ce3c.slice/crio-21b1e53cea20c2e5ae05aa239283c9a0c77ff9db1ea83f58fd74cc5fc119cdb2 WatchSource:0}: Error finding container 21b1e53cea20c2e5ae05aa239283c9a0c77ff9db1ea83f58fd74cc5fc119cdb2: Status 404 returned error can't find the container with id 21b1e53cea20c2e5ae05aa239283c9a0c77ff9db1ea83f58fd74cc5fc119cdb2 Jul 06 19:36:38.322925 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.322872 2577 generic.go:356] "Generic (PLEG): container finished" podID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" containerID="0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94" exitCode=0 Jul 06 19:36:38.323324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.322928 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" Jul 06 19:36:38.323324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.322957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" event={"ID":"f0ed648c-1d43-4a87-ae47-ab6e6829f84a","Type":"ContainerDied","Data":"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94"} Jul 06 19:36:38.323324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.323005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz" event={"ID":"f0ed648c-1d43-4a87-ae47-ab6e6829f84a","Type":"ContainerDied","Data":"67b5c2153d6f84bcb43c092a4908d1c64297c9735af49e3b218fb08937e3eeaa"} Jul 06 19:36:38.323324 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.323033 2577 scope.go:117] "RemoveContainer" containerID="0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94" Jul 06 19:36:38.324675 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.324641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" event={"ID":"ae9458a0-518c-4292-bf9f-d6ed6937ce3c","Type":"ContainerStarted","Data":"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07"} Jul 06 19:36:38.324782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.324690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" event={"ID":"ae9458a0-518c-4292-bf9f-d6ed6937ce3c","Type":"ContainerStarted","Data":"21b1e53cea20c2e5ae05aa239283c9a0c77ff9db1ea83f58fd74cc5fc119cdb2"} Jul 06 19:36:38.324782 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.324722 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:38.331932 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.331914 2577 scope.go:117] "RemoveContainer" containerID="0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94" Jul 06 19:36:38.332170 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:38.332149 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94\": container with ID starting with 0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94 not found: ID does not exist" containerID="0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94" Jul 06 19:36:38.332229 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.332182 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94"} err="failed to get container status \"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94\": rpc error: code = NotFound desc = could not find container \"0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94\": container with ID starting with 0de272404f78bcbc5ddf0e02e07b7af4f32c6cd8ab474bded4144f157ea2cb94 not found: ID does not exist" Jul 06 19:36:38.358150 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.358129 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:38.363393 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.363372 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-p9khz"] Jul 06 19:36:38.400706 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.400668 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" podStartSLOduration=1.4006556159999999 podStartE2EDuration="1.400655616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:38.396824163 +0000 UTC m=+556.050391761" watchObservedRunningTime="2026-07-06 19:36:38.400655616 +0000 UTC m=+556.054223214" Jul 06 19:36:38.809501 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:38.809470 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" path="/var/lib/kubelet/pods/f0ed648c-1d43-4a87-ae47-ab6e6829f84a/volumes" Jul 06 19:36:49.333180 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.333151 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:36:49.394428 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.394398 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:49.394621 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.394584 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" podUID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" containerName="manager" containerID="cri-o://09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437" gracePeriod=10 Jul 06 19:36:49.666570 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.666528 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:49.732653 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.732626 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume\") pod \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " Jul 06 19:36:49.732777 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.732706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9l8d\" (UniqueName: \"kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d\") pod \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\" (UID: \"d08d40dd-0f22-4ed4-8c88-ce285cca11c0\") " Jul 06 19:36:49.733017 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.732995 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d08d40dd-0f22-4ed4-8c88-ce285cca11c0" (UID: "d08d40dd-0f22-4ed4-8c88-ce285cca11c0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:36:49.734594 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.734572 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d" (OuterVolumeSpecName: "kube-api-access-w9l8d") pod "d08d40dd-0f22-4ed4-8c88-ce285cca11c0" (UID: "d08d40dd-0f22-4ed4-8c88-ce285cca11c0"). InnerVolumeSpecName "kube-api-access-w9l8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:49.834206 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.834175 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9l8d\" (UniqueName: \"kubernetes.io/projected/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-kube-api-access-w9l8d\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:49.834206 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:49.834201 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d08d40dd-0f22-4ed4-8c88-ce285cca11c0-extensions-socket-volume\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:36:50.391051 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.391014 2577 generic.go:356] "Generic (PLEG): container finished" podID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" containerID="09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437" exitCode=0 Jul 06 19:36:50.391440 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.391060 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" Jul 06 19:36:50.391440 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.391100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" event={"ID":"d08d40dd-0f22-4ed4-8c88-ce285cca11c0","Type":"ContainerDied","Data":"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437"} Jul 06 19:36:50.391440 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.391157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf" event={"ID":"d08d40dd-0f22-4ed4-8c88-ce285cca11c0","Type":"ContainerDied","Data":"65f35351ae7cb241714a32af706f19688f7d5096e963d53a7ba410844b385a66"} Jul 06 19:36:50.391440 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.391184 2577 scope.go:117] "RemoveContainer" containerID="09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437" Jul 06 19:36:50.401069 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.400911 2577 scope.go:117] "RemoveContainer" containerID="09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437" Jul 06 19:36:50.401180 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:36:50.401160 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437\": container with ID starting with 09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437 not found: ID does not exist" containerID="09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437" Jul 06 19:36:50.401221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.401190 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437"} err="failed to get container status \"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437\": rpc error: code = NotFound desc = could not find container \"09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437\": container with ID starting with 09bdff6d99016c5e0365e1340c97e75cc81c47dddbb5f11a0eaaa80f3f06b437 not found: ID does not exist" Jul 06 19:36:50.416482 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.416457 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:50.422531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.422512 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tv8kf"] Jul 06 19:36:50.809592 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:50.809564 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" path="/var/lib/kubelet/pods/d08d40dd-0f22-4ed4-8c88-ce285cca11c0/volumes" Jul 06 19:36:53.753000 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:53.752961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:53.998285 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:36:53.998255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:07.904988 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.904952 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:07.905895 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.905873 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:07.906253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906239 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" containerName="manager" Jul 06 19:37:07.906291 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906255 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" containerName="manager" Jul 06 19:37:07.906291 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906267 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" containerName="manager" Jul 06 19:37:07.906291 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906275 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" containerName="manager" Jul 06 19:37:07.906383 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906342 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ed648c-1d43-4a87-ae47-ab6e6829f84a" containerName="manager" Jul 06 19:37:07.906383 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.906355 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08d40dd-0f22-4ed4-8c88-ce285cca11c0" containerName="manager" Jul 06 19:37:07.964531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.964476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:07.964697 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.964607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:07.967531 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:07.967509 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:37:08.001609 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.001570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:08.010347 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.010319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:08.073047 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.073018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.073167 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.073067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7rq\" (UniqueName: \"kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.108037 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.108008 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:08.174269 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.174187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7rq\" (UniqueName: \"kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.174385 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.174269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.174931 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.174914 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.182703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.182682 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7rq\" (UniqueName: \"kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq\") pod \"limitador-limitador-64c94fd64f-tdfm6\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.277127 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.277096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:08.414448 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.414425 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:08.416138 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:08.416108 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ceffec_3509_4016_a8ff_3ec54e41e4d2.slice/crio-b2d1aaa83eb425877c057bc72bc10c754e8f6e6e064b9846dc5008f54e1ac45a WatchSource:0}: Error finding container b2d1aaa83eb425877c057bc72bc10c754e8f6e6e064b9846dc5008f54e1ac45a: Status 404 returned error can't find the container with id b2d1aaa83eb425877c057bc72bc10c754e8f6e6e064b9846dc5008f54e1ac45a Jul 06 19:37:08.485411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.485381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" event={"ID":"11ceffec-3509-4016-a8ff-3ec54e41e4d2","Type":"ContainerStarted","Data":"b2d1aaa83eb425877c057bc72bc10c754e8f6e6e064b9846dc5008f54e1ac45a"} Jul 06 19:37:08.754926 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.754838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:08.841973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.841938 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:08.843086 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.843066 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:08.847908 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.847876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:08.850811 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.850789 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kts8m\"" Jul 06 19:37:08.852141 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.852116 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:08.981743 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:08.981712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrcq\" (UniqueName: \"kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq\") pod \"authorino-75d556bd77-rhmph\" (UID: \"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d\") " pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:09.082922 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:09.082830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrcq\" (UniqueName: \"kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq\") pod \"authorino-75d556bd77-rhmph\" (UID: \"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d\") " pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:09.092696 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:09.092671 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrcq\" (UniqueName: \"kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq\") pod \"authorino-75d556bd77-rhmph\" (UID: \"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d\") " pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:09.157932 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:09.157892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:09.288022 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:09.287972 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:09.290319 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:09.290288 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc02e6ad_88d7_43a5_8e70_13b18cf5ea0d.slice/crio-fd29e10fe262fc545a1aec4fa399e56f7b309d52503304a6b2743e4150b92fe5 WatchSource:0}: Error finding container fd29e10fe262fc545a1aec4fa399e56f7b309d52503304a6b2743e4150b92fe5: Status 404 returned error can't find the container with id fd29e10fe262fc545a1aec4fa399e56f7b309d52503304a6b2743e4150b92fe5 Jul 06 19:37:09.497237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:09.497194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rhmph" event={"ID":"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d","Type":"ContainerStarted","Data":"fd29e10fe262fc545a1aec4fa399e56f7b309d52503304a6b2743e4150b92fe5"} Jul 06 19:37:13.529881 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:13.529851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rhmph" event={"ID":"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d","Type":"ContainerStarted","Data":"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8"} Jul 06 19:37:13.531526 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:13.531495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" event={"ID":"11ceffec-3509-4016-a8ff-3ec54e41e4d2","Type":"ContainerStarted","Data":"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b"} Jul 06 19:37:13.531646 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:13.531548 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:13.545932 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:13.545871 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-rhmph" podStartSLOduration=1.628019101 podStartE2EDuration="5.545857322s" totalImagesPullingTime="3.917838221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:13.54507485 +0000 UTC m=+591.198642449" watchObservedRunningTime="2026-07-06 19:37:13.545857322 +0000 UTC m=+591.199424922" Jul 06 19:37:13.564286 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:13.564239 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" podStartSLOduration=1.785876242 podStartE2EDuration="6.564223148s" totalImagesPullingTime="4.778346906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:13.561688565 +0000 UTC m=+591.215256164" watchObservedRunningTime="2026-07-06 19:37:13.564223148 +0000 UTC m=+591.217790757" Jul 06 19:37:23.099261 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:23.099181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:24.538211 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:24.538182 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:24.726434 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:24.726400 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:24.726606 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:24.726582 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" podUID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" containerName="limitador" containerID="cri-o://8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b" gracePeriod=30 Jul 06 19:37:25.267938 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.267893 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:25.410568 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.410485 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm7rq\" (UniqueName: \"kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq\") pod \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " Jul 06 19:37:25.410568 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.410566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file\") pod \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\" (UID: \"11ceffec-3509-4016-a8ff-3ec54e41e4d2\") " Jul 06 19:37:25.410951 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.410926 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file" (OuterVolumeSpecName: "config-file") pod "11ceffec-3509-4016-a8ff-3ec54e41e4d2" (UID: "11ceffec-3509-4016-a8ff-3ec54e41e4d2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:37:25.412606 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.412572 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq" (OuterVolumeSpecName: "kube-api-access-pm7rq") pod "11ceffec-3509-4016-a8ff-3ec54e41e4d2" (UID: "11ceffec-3509-4016-a8ff-3ec54e41e4d2"). InnerVolumeSpecName "kube-api-access-pm7rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:37:25.511055 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.511028 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7rq\" (UniqueName: \"kubernetes.io/projected/11ceffec-3509-4016-a8ff-3ec54e41e4d2-kube-api-access-pm7rq\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:37:25.511055 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.511051 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/11ceffec-3509-4016-a8ff-3ec54e41e4d2-config-file\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:37:25.600285 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.600259 2577 generic.go:356] "Generic (PLEG): container finished" podID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" containerID="8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b" exitCode=0 Jul 06 19:37:25.600580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.600323 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" Jul 06 19:37:25.600580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.600347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" event={"ID":"11ceffec-3509-4016-a8ff-3ec54e41e4d2","Type":"ContainerDied","Data":"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b"} Jul 06 19:37:25.600580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.600398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tdfm6" event={"ID":"11ceffec-3509-4016-a8ff-3ec54e41e4d2","Type":"ContainerDied","Data":"b2d1aaa83eb425877c057bc72bc10c754e8f6e6e064b9846dc5008f54e1ac45a"} Jul 06 19:37:25.600580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.600422 2577 scope.go:117] "RemoveContainer" containerID="8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b" Jul 06 19:37:25.609997 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.609983 2577 scope.go:117] "RemoveContainer" containerID="8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b" Jul 06 19:37:25.610220 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:37:25.610203 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b\": container with ID starting with 8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b not found: ID does not exist" containerID="8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b" Jul 06 19:37:25.610276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.610232 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b"} err="failed to get container status \"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b\": rpc error: code = NotFound desc = could not find container \"8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b\": container with ID starting with 8e6ceef4c1e6e2412b1087bcc87e4929c07f8af7bf5c8656cad4993e12e12b8b not found: ID does not exist" Jul 06 19:37:25.624211 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.624191 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:25.628362 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:25.628340 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tdfm6"] Jul 06 19:37:26.809000 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:26.808972 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" path="/var/lib/kubelet/pods/11ceffec-3509-4016-a8ff-3ec54e41e4d2/volumes" Jul 06 19:37:27.204628 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.204553 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-fh2jn"] Jul 06 19:37:27.204970 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.204953 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" containerName="limitador" Jul 06 19:37:27.204970 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.204969 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" containerName="limitador" Jul 06 19:37:27.205117 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.205034 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ceffec-3509-4016-a8ff-3ec54e41e4d2" containerName="limitador" Jul 06 19:37:27.208927 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.208894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.212104 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.212085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:37:27.212202 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.212136 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p26gs\"" Jul 06 19:37:27.220051 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.220027 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-fh2jn"] Jul 06 19:37:27.243810 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.243787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:27.323172 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.323142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4143ea7c-27eb-490c-83a1-fd6b025e6c38-data\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.323315 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.323185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbr8f\" (UniqueName: \"kubernetes.io/projected/4143ea7c-27eb-490c-83a1-fd6b025e6c38-kube-api-access-fbr8f\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.424437 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.424402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbr8f\" (UniqueName: \"kubernetes.io/projected/4143ea7c-27eb-490c-83a1-fd6b025e6c38-kube-api-access-fbr8f\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.424540 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.424485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4143ea7c-27eb-490c-83a1-fd6b025e6c38-data\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.424799 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.424783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4143ea7c-27eb-490c-83a1-fd6b025e6c38-data\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.434349 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.434319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbr8f\" (UniqueName: \"kubernetes.io/projected/4143ea7c-27eb-490c-83a1-fd6b025e6c38-kube-api-access-fbr8f\") pod \"postgres-69fb4594cf-fh2jn\" (UID: \"4143ea7c-27eb-490c-83a1-fd6b025e6c38\") " pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.520799 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.520778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:27.647072 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:27.647048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-fh2jn"] Jul 06 19:37:27.649206 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:27.649181 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4143ea7c_27eb_490c_83a1_fd6b025e6c38.slice/crio-4c53f343fff6d5c18e4d8b49f9e1df0832cfed4fa6e31ac9f05a7a5b3051ff12 WatchSource:0}: Error finding container 4c53f343fff6d5c18e4d8b49f9e1df0832cfed4fa6e31ac9f05a7a5b3051ff12: Status 404 returned error can't find the container with id 4c53f343fff6d5c18e4d8b49f9e1df0832cfed4fa6e31ac9f05a7a5b3051ff12 Jul 06 19:37:28.622678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:28.622604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-fh2jn" event={"ID":"4143ea7c-27eb-490c-83a1-fd6b025e6c38","Type":"ContainerStarted","Data":"4c53f343fff6d5c18e4d8b49f9e1df0832cfed4fa6e31ac9f05a7a5b3051ff12"} Jul 06 19:37:33.652381 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:33.652344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-fh2jn" event={"ID":"4143ea7c-27eb-490c-83a1-fd6b025e6c38","Type":"ContainerStarted","Data":"611d07f4833e5de1fced0794161093d8cbb811a112e7d1fefb42ec4a9824cf00"} Jul 06 19:37:33.652762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:33.652431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:33.670650 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:33.670594 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-fh2jn" podStartSLOduration=1.186169302 podStartE2EDuration="6.670572645s" totalImagesPullingTime="5.484403343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:33.668632722 +0000 UTC m=+611.322200321" watchObservedRunningTime="2026-07-06 19:37:33.670572645 +0000 UTC m=+611.324140245" Jul 06 19:37:39.689707 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:39.689673 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-fh2jn" Jul 06 19:37:40.023245 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.023196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:40.523309 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.523276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:40.533858 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.533829 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:40.539497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.539475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:40.546571 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.546536 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:40.637154 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.637129 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6p2c\" (UniqueName: \"kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c\") pod \"authorino-866b77749d-4dvk6\" (UID: \"296af8ee-2db9-4d55-ad2d-2bb332771edf\") " pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:40.737448 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.737422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p2c\" (UniqueName: \"kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c\") pod \"authorino-866b77749d-4dvk6\" (UID: \"296af8ee-2db9-4d55-ad2d-2bb332771edf\") " pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:40.745545 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.745523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p2c\" (UniqueName: \"kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c\") pod \"authorino-866b77749d-4dvk6\" (UID: \"296af8ee-2db9-4d55-ad2d-2bb332771edf\") " pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:40.786279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.786213 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:40.786430 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.786419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:40.918671 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:40.918644 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:40.920580 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:40.920551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod296af8ee_2db9_4d55_ad2d_2bb332771edf.slice/crio-941468d650b482d8bb74fff53938c1d2d3853f212212fd6f07c166a18ab3acfe WatchSource:0}: Error finding container 941468d650b482d8bb74fff53938c1d2d3853f212212fd6f07c166a18ab3acfe: Status 404 returned error can't find the container with id 941468d650b482d8bb74fff53938c1d2d3853f212212fd6f07c166a18ab3acfe Jul 06 19:37:41.696958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.696857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866b77749d-4dvk6" event={"ID":"296af8ee-2db9-4d55-ad2d-2bb332771edf","Type":"ContainerStarted","Data":"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f"} Jul 06 19:37:41.696958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.696918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866b77749d-4dvk6" event={"ID":"296af8ee-2db9-4d55-ad2d-2bb332771edf","Type":"ContainerStarted","Data":"941468d650b482d8bb74fff53938c1d2d3853f212212fd6f07c166a18ab3acfe"} Jul 06 19:37:41.697146 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.696955 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-866b77749d-4dvk6" podUID="296af8ee-2db9-4d55-ad2d-2bb332771edf" containerName="authorino" containerID="cri-o://4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f" gracePeriod=30 Jul 06 19:37:41.836678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.836612 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-866b77749d-4dvk6" podStartSLOduration=1.5284838710000002 podStartE2EDuration="1.836588279s" totalImagesPullingTime="308.104408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:41.714146451 +0000 UTC m=+619.367714051" watchObservedRunningTime="2026-07-06 19:37:41.836588279 +0000 UTC m=+619.490155880" Jul 06 19:37:41.837568 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.837547 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:41.837785 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.837759 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-rhmph" podUID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" containerName="authorino" containerID="cri-o://bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8" gracePeriod=30 Jul 06 19:37:41.961962 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:41.961942 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:42.048371 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.048343 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6p2c\" (UniqueName: \"kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c\") pod \"296af8ee-2db9-4d55-ad2d-2bb332771edf\" (UID: \"296af8ee-2db9-4d55-ad2d-2bb332771edf\") " Jul 06 19:37:42.050477 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.050446 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c" (OuterVolumeSpecName: "kube-api-access-z6p2c") pod "296af8ee-2db9-4d55-ad2d-2bb332771edf" (UID: "296af8ee-2db9-4d55-ad2d-2bb332771edf"). InnerVolumeSpecName "kube-api-access-z6p2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:37:42.090958 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.090935 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:42.149671 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.149645 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptrcq\" (UniqueName: \"kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq\") pod \"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d\" (UID: \"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d\") " Jul 06 19:37:42.149987 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.149968 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6p2c\" (UniqueName: \"kubernetes.io/projected/296af8ee-2db9-4d55-ad2d-2bb332771edf-kube-api-access-z6p2c\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:37:42.151600 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.151579 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq" (OuterVolumeSpecName: "kube-api-access-ptrcq") pod "dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" (UID: "dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d"). InnerVolumeSpecName "kube-api-access-ptrcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:37:42.250684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.250652 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrcq\" (UniqueName: \"kubernetes.io/projected/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d-kube-api-access-ptrcq\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:37:42.702789 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.702702 2577 generic.go:356] "Generic (PLEG): container finished" podID="296af8ee-2db9-4d55-ad2d-2bb332771edf" containerID="4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f" exitCode=0 Jul 06 19:37:42.702789 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.702740 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866b77749d-4dvk6" Jul 06 19:37:42.702789 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.702769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866b77749d-4dvk6" event={"ID":"296af8ee-2db9-4d55-ad2d-2bb332771edf","Type":"ContainerDied","Data":"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f"} Jul 06 19:37:42.703049 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.702806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866b77749d-4dvk6" event={"ID":"296af8ee-2db9-4d55-ad2d-2bb332771edf","Type":"ContainerDied","Data":"941468d650b482d8bb74fff53938c1d2d3853f212212fd6f07c166a18ab3acfe"} Jul 06 19:37:42.703049 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.702822 2577 scope.go:117] "RemoveContainer" containerID="4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f" Jul 06 19:37:42.704124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.704101 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" containerID="bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8" exitCode=0 Jul 06 19:37:42.704231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.704134 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rhmph" Jul 06 19:37:42.704231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.704178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rhmph" event={"ID":"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d","Type":"ContainerDied","Data":"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8"} Jul 06 19:37:42.704231 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.704221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rhmph" event={"ID":"dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d","Type":"ContainerDied","Data":"fd29e10fe262fc545a1aec4fa399e56f7b309d52503304a6b2743e4150b92fe5"} Jul 06 19:37:42.713241 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.713225 2577 scope.go:117] "RemoveContainer" containerID="4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f" Jul 06 19:37:42.713523 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:37:42.713506 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f\": container with ID starting with 4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f not found: ID does not exist" containerID="4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f" Jul 06 19:37:42.713613 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.713534 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f"} err="failed to get container status \"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f\": rpc error: code = NotFound desc = could not find container \"4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f\": container with ID starting with 4bbaf47e5d66ebac9bf99e3399867397995dd1c41b02f3ee967626727b20aa4f not found: ID does not exist" Jul 06 19:37:42.713613 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.713555 2577 scope.go:117] "RemoveContainer" containerID="bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8" Jul 06 19:37:42.721084 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.721069 2577 scope.go:117] "RemoveContainer" containerID="bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8" Jul 06 19:37:42.721313 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:37:42.721295 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8\": container with ID starting with bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8 not found: ID does not exist" containerID="bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8" Jul 06 19:37:42.721368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.721323 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8"} err="failed to get container status \"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8\": rpc error: code = NotFound desc = could not find container \"bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8\": container with ID starting with bdac59f2bc813b5a85cc6cfa09c96b0f68b822cc6c9aa32beed02f4140ba5ae8 not found: ID does not exist" Jul 06 19:37:42.727013 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.726993 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:42.731630 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.731611 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-866b77749d-4dvk6"] Jul 06 19:37:42.744341 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.744322 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:42.751841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.751824 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rhmph"] Jul 06 19:37:42.809236 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.809217 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296af8ee-2db9-4d55-ad2d-2bb332771edf" path="/var/lib/kubelet/pods/296af8ee-2db9-4d55-ad2d-2bb332771edf/volumes" Jul 06 19:37:42.809529 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:42.809516 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" path="/var/lib/kubelet/pods/dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d/volumes" Jul 06 19:37:46.309655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.309567 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.329988 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296af8ee-2db9-4d55-ad2d-2bb332771edf" containerName="authorino" Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330470 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="296af8ee-2db9-4d55-ad2d-2bb332771edf" containerName="authorino" Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330487 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" containerName="authorino" Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330495 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" containerName="authorino" Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330581 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="296af8ee-2db9-4d55-ad2d-2bb332771edf" containerName="authorino" Jul 06 19:37:46.401974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.330594 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc02e6ad-88d7-43a5-8e70-13b18cf5ea0d" containerName="authorino" Jul 06 19:37:46.410217 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.410194 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:37:46.410296 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.410289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.413330 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.413310 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:37:46.413468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.413339 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:37:46.413536 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.413521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-86whh\"" Jul 06 19:37:46.486101 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.486066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.486237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.486180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmd9\" (UniqueName: \"kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.532623 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.532597 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:37:46.563255 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.563200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:37:46.563255 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.563214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.586684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.586661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmd9\" (UniqueName: \"kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.586778 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.586735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.589145 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.589122 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.596381 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.596359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmd9\" (UniqueName: \"kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9\") pod \"maas-controller-7ffd75f595-cf9wd\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.687729 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.687704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7n9\" (UniqueName: \"kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.687827 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.687742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.720999 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.720976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:46.788485 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.788452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7n9\" (UniqueName: \"kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.788625 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.788520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.791613 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.791560 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.797214 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.797196 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7n9\" (UniqueName: \"kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9\") pod \"maas-controller-84b8f5864f-7wkvh\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.872699 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.872674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:46.876157 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.876136 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:37:46.878477 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:46.878452 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160efea7_7129_4aaf_a14e_ac07e06ab06d.slice/crio-34ded95fd9c0299a776e7c4ed72c57d939fbfdf1a1a8c3b1083db1054dcd035a WatchSource:0}: Error finding container 34ded95fd9c0299a776e7c4ed72c57d939fbfdf1a1a8c3b1083db1054dcd035a: Status 404 returned error can't find the container with id 34ded95fd9c0299a776e7c4ed72c57d939fbfdf1a1a8c3b1083db1054dcd035a Jul 06 19:37:46.879869 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:46.879854 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:37:47.063144 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:47.063119 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:37:47.063769 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:37:47.063742 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908560d8_23e5_43be_a386_a3b3e99824d6.slice/crio-8f7b6d3bdfce0bd68a8080df7d6c358e81e34fae9f37261269f9e85efc3a0d8e WatchSource:0}: Error finding container 8f7b6d3bdfce0bd68a8080df7d6c358e81e34fae9f37261269f9e85efc3a0d8e: Status 404 returned error can't find the container with id 8f7b6d3bdfce0bd68a8080df7d6c358e81e34fae9f37261269f9e85efc3a0d8e Jul 06 19:37:47.380040 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:47.380004 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:47.740238 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:47.740084 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" event={"ID":"908560d8-23e5-43be-a386-a3b3e99824d6","Type":"ContainerStarted","Data":"8f7b6d3bdfce0bd68a8080df7d6c358e81e34fae9f37261269f9e85efc3a0d8e"} Jul 06 19:37:47.742456 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:47.742403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" event={"ID":"160efea7-7129-4aaf-a14e-ac07e06ab06d","Type":"ContainerStarted","Data":"34ded95fd9c0299a776e7c4ed72c57d939fbfdf1a1a8c3b1083db1054dcd035a"} Jul 06 19:37:49.756557 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.756526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" event={"ID":"908560d8-23e5-43be-a386-a3b3e99824d6","Type":"ContainerStarted","Data":"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff"} Jul 06 19:37:49.756995 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.756650 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:37:49.758422 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.758386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" event={"ID":"160efea7-7129-4aaf-a14e-ac07e06ab06d","Type":"ContainerStarted","Data":"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92"} Jul 06 19:37:49.758523 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.758442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:37:49.776566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.776524 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" podStartSLOduration=1.233806783 podStartE2EDuration="3.776509598s" totalImagesPullingTime="2.542702815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:49.775371026 +0000 UTC m=+627.428938626" watchObservedRunningTime="2026-07-06 19:37:49.776509598 +0000 UTC m=+627.430077196" Jul 06 19:37:49.794367 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.794309 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" podStartSLOduration=1.058485534 podStartE2EDuration="3.794288544s" totalImagesPullingTime="2.73580301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:49.792228355 +0000 UTC m=+627.445795956" watchObservedRunningTime="2026-07-06 19:37:49.794288544 +0000 UTC m=+627.447856143" Jul 06 19:37:49.889368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.889268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:49.975788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:37:49.975755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:00.770818 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:00.770786 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:38:00.771223 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:00.771080 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:38:00.835662 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:00.835621 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:38:00.835839 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:00.835802 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" podUID="160efea7-7129-4aaf-a14e-ac07e06ab06d" containerName="manager" containerID="cri-o://2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92" gracePeriod=30 Jul 06 19:38:01.076625 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.076605 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:38:01.204133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.204105 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert\") pod \"160efea7-7129-4aaf-a14e-ac07e06ab06d\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " Jul 06 19:38:01.204276 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.204167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgmd9\" (UniqueName: \"kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9\") pod \"160efea7-7129-4aaf-a14e-ac07e06ab06d\" (UID: \"160efea7-7129-4aaf-a14e-ac07e06ab06d\") " Jul 06 19:38:01.206306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.206272 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9" (OuterVolumeSpecName: "kube-api-access-dgmd9") pod "160efea7-7129-4aaf-a14e-ac07e06ab06d" (UID: "160efea7-7129-4aaf-a14e-ac07e06ab06d"). InnerVolumeSpecName "kube-api-access-dgmd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:01.206306 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.206288 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "160efea7-7129-4aaf-a14e-ac07e06ab06d" (UID: "160efea7-7129-4aaf-a14e-ac07e06ab06d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:01.304906 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.304868 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/160efea7-7129-4aaf-a14e-ac07e06ab06d-webhook-cert\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:38:01.305033 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.304921 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmd9\" (UniqueName: \"kubernetes.io/projected/160efea7-7129-4aaf-a14e-ac07e06ab06d-kube-api-access-dgmd9\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:38:01.820274 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.820242 2577 generic.go:356] "Generic (PLEG): container finished" podID="160efea7-7129-4aaf-a14e-ac07e06ab06d" containerID="2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92" exitCode=0 Jul 06 19:38:01.820673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.820282 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" Jul 06 19:38:01.820673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.820328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" event={"ID":"160efea7-7129-4aaf-a14e-ac07e06ab06d","Type":"ContainerDied","Data":"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92"} Jul 06 19:38:01.820673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.820376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ffd75f595-cf9wd" event={"ID":"160efea7-7129-4aaf-a14e-ac07e06ab06d","Type":"ContainerDied","Data":"34ded95fd9c0299a776e7c4ed72c57d939fbfdf1a1a8c3b1083db1054dcd035a"} Jul 06 19:38:01.820673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.820399 2577 scope.go:117] "RemoveContainer" containerID="2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92" Jul 06 19:38:01.829566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.829548 2577 scope.go:117] "RemoveContainer" containerID="2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92" Jul 06 19:38:01.829853 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:38:01.829836 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92\": container with ID starting with 2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92 not found: ID does not exist" containerID="2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92" Jul 06 19:38:01.829940 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.829859 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92"} err="failed to get container status \"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92\": rpc error: code = NotFound desc = could not find container \"2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92\": container with ID starting with 2189710683c0f27801e94fff104078e11c0b80446e19b424b7580c761c292c92 not found: ID does not exist" Jul 06 19:38:01.844968 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.844944 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:38:01.849737 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:01.849718 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ffd75f595-cf9wd"] Jul 06 19:38:02.809256 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:02.809226 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160efea7-7129-4aaf-a14e-ac07e06ab06d" path="/var/lib/kubelet/pods/160efea7-7129-4aaf-a14e-ac07e06ab06d/volumes" Jul 06 19:38:15.527474 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:15.527439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:18.464441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.464399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:18.954626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.954594 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:18.992833 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.992804 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-7c874b949-m545l"] Jul 06 19:38:18.993239 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.993221 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160efea7-7129-4aaf-a14e-ac07e06ab06d" containerName="manager" Jul 06 19:38:18.993298 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.993242 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="160efea7-7129-4aaf-a14e-ac07e06ab06d" containerName="manager" Jul 06 19:38:18.993356 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.993344 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="160efea7-7129-4aaf-a14e-ac07e06ab06d" containerName="manager" Jul 06 19:38:18.996238 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.996224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:18.999317 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:18.999297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 19:38:19.015287 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.012090 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7c874b949-m545l"] Jul 06 19:38:19.086396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.086371 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:19.150843 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.150816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-maas-api-tls\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.150994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.150859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-openshift-service-ca\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.150994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.150878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhd6n\" (UniqueName: \"kubernetes.io/projected/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-kube-api-access-xhd6n\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.252183 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.252155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-maas-api-tls\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.252327 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.252202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-openshift-service-ca\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.252327 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.252229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhd6n\" (UniqueName: \"kubernetes.io/projected/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-kube-api-access-xhd6n\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.252929 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.252890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-openshift-service-ca\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.254546 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.254524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-maas-api-tls\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.261377 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.261350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhd6n\" (UniqueName: \"kubernetes.io/projected/1a5c315d-e4df-4aff-87d4-cd288dc5aa95-kube-api-access-xhd6n\") pod \"maas-api-7c874b949-m545l\" (UID: \"1a5c315d-e4df-4aff-87d4-cd288dc5aa95\") " pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.306320 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.306284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:19.447126 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.447085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:19.463074 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.462860 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7c874b949-m545l"] Jul 06 19:38:19.465560 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:38:19.465534 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5c315d_e4df_4aff_87d4_cd288dc5aa95.slice/crio-8f4d6cdc533eb30dbe375c7f80d61f4ecf3535d38b097d351af864009c1d6907 WatchSource:0}: Error finding container 8f4d6cdc533eb30dbe375c7f80d61f4ecf3535d38b097d351af864009c1d6907: Status 404 returned error can't find the container with id 8f4d6cdc533eb30dbe375c7f80d61f4ecf3535d38b097d351af864009c1d6907 Jul 06 19:38:19.536102 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.536026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:19.571760 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.571736 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-hr9tb"] Jul 06 19:38:19.576317 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.576301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.579332 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.579315 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b82g9\"" Jul 06 19:38:19.579435 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.579316 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 19:38:19.587153 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.587124 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-hr9tb"] Jul 06 19:38:19.757020 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.756986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptdn\" (UniqueName: \"kubernetes.io/projected/443768c2-5343-482b-ad62-77576ef6d2aa-kube-api-access-hptdn\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.757169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.757026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443768c2-5343-482b-ad62-77576ef6d2aa-plugins-config-volume\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.858274 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.858196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptdn\" (UniqueName: \"kubernetes.io/projected/443768c2-5343-482b-ad62-77576ef6d2aa-kube-api-access-hptdn\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.858274 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.858233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443768c2-5343-482b-ad62-77576ef6d2aa-plugins-config-volume\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.859006 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.858988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443768c2-5343-482b-ad62-77576ef6d2aa-plugins-config-volume\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.867317 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.867293 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptdn\" (UniqueName: \"kubernetes.io/projected/443768c2-5343-482b-ad62-77576ef6d2aa-kube-api-access-hptdn\") pod \"payload-processing-58f86c7798-hr9tb\" (UID: \"443768c2-5343-482b-ad62-77576ef6d2aa\") " pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.888197 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.888174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:19.913951 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:19.913920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7c874b949-m545l" event={"ID":"1a5c315d-e4df-4aff-87d4-cd288dc5aa95","Type":"ContainerStarted","Data":"8f4d6cdc533eb30dbe375c7f80d61f4ecf3535d38b097d351af864009c1d6907"} Jul 06 19:38:20.021286 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:20.021250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-hr9tb"] Jul 06 19:38:20.023818 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:38:20.023792 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443768c2_5343_482b_ad62_77576ef6d2aa.slice/crio-50014b1ffab513d4ebb5307c3efb40dd16f658cde23ac9000b21e07a18d997ff WatchSource:0}: Error finding container 50014b1ffab513d4ebb5307c3efb40dd16f658cde23ac9000b21e07a18d997ff: Status 404 returned error can't find the container with id 50014b1ffab513d4ebb5307c3efb40dd16f658cde23ac9000b21e07a18d997ff Jul 06 19:38:20.924427 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:20.924372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" event={"ID":"443768c2-5343-482b-ad62-77576ef6d2aa","Type":"ContainerStarted","Data":"50014b1ffab513d4ebb5307c3efb40dd16f658cde23ac9000b21e07a18d997ff"} Jul 06 19:38:21.931394 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.931316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" event={"ID":"443768c2-5343-482b-ad62-77576ef6d2aa","Type":"ContainerStarted","Data":"f4481cbed650b191a3ca4cbbf0017e65590f2f30c41b91455888a970e7a588d2"} Jul 06 19:38:21.931821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.931414 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:21.932786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.932762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7c874b949-m545l" event={"ID":"1a5c315d-e4df-4aff-87d4-cd288dc5aa95","Type":"ContainerStarted","Data":"10b416578980a4230ec1a573967b430fbf453e8dc1000f872791ebcef04a0667"} Jul 06 19:38:21.932939 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.932813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:21.951999 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.951953 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" podStartSLOduration=1.345978308 podStartE2EDuration="2.951939811s" totalImagesPullingTime="1.605961503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:21.949188319 +0000 UTC m=+659.602755917" watchObservedRunningTime="2026-07-06 19:38:21.951939811 +0000 UTC m=+659.605507410" Jul 06 19:38:21.972418 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:21.972376 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-7c874b949-m545l" podStartSLOduration=1.846721174 podStartE2EDuration="3.972364004s" totalImagesPullingTime="2.12564283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:21.970786548 +0000 UTC m=+659.624354147" watchObservedRunningTime="2026-07-06 19:38:21.972364004 +0000 UTC m=+659.625931602" Jul 06 19:38:27.942780 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:27.942753 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-7c874b949-m545l" Jul 06 19:38:28.951558 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:28.951529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:29.050959 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:29.050930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:32.939802 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:32.939766 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-hr9tb" Jul 06 19:38:34.296218 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.296186 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w"] Jul 06 19:38:34.300161 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.300144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.303078 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.303056 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fkdk9\"" Jul 06 19:38:34.303240 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.303216 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 19:38:34.304515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.304498 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:38:34.304985 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.304968 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:38:34.312841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.312818 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w"] Jul 06 19:38:34.339038 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.339003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:34.368842 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.368817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.368978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.368857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.368978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.368885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxk2\" (UniqueName: \"kubernetes.io/projected/f983db50-3c59-43a9-a64f-18912819a236-kube-api-access-5fxk2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.369090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.368984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.369090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.369014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f983db50-3c59-43a9-a64f-18912819a236-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.369090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.369048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470253 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxk2\" (UniqueName: \"kubernetes.io/projected/f983db50-3c59-43a9-a64f-18912819a236-kube-api-access-5fxk2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f983db50-3c59-43a9-a64f-18912819a236-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470621 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470674 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470769 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.470827 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.470800 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.472573 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.472548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f983db50-3c59-43a9-a64f-18912819a236-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.472762 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.472742 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f983db50-3c59-43a9-a64f-18912819a236-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.480476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.480451 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxk2\" (UniqueName: \"kubernetes.io/projected/f983db50-3c59-43a9-a64f-18912819a236-kube-api-access-5fxk2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w\" (UID: \"f983db50-3c59-43a9-a64f-18912819a236\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.611518 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.611459 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:34.750055 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:34.750030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w"] Jul 06 19:38:34.751991 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:38:34.751961 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf983db50_3c59_43a9_a64f_18912819a236.slice/crio-2f9b103ec323cb272108c8361d8cfad84b028d44095dac803157d38ab572e8cb WatchSource:0}: Error finding container 2f9b103ec323cb272108c8361d8cfad84b028d44095dac803157d38ab572e8cb: Status 404 returned error can't find the container with id 2f9b103ec323cb272108c8361d8cfad84b028d44095dac803157d38ab572e8cb Jul 06 19:38:35.006225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:35.006117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" event={"ID":"f983db50-3c59-43a9-a64f-18912819a236","Type":"ContainerStarted","Data":"2f9b103ec323cb272108c8361d8cfad84b028d44095dac803157d38ab572e8cb"} Jul 06 19:38:36.012880 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:36.012798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" event={"ID":"f983db50-3c59-43a9-a64f-18912819a236","Type":"ContainerStarted","Data":"7cca459cad4d610c744b485ac4f6da9d8d584b6b4e9742043f1b991ece5512cc"} Jul 06 19:38:36.033988 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:36.033926 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" podStartSLOduration=1.029424572 podStartE2EDuration="2.03388086s" totalImagesPullingTime="1.004456288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:36.031976744 +0000 UTC m=+673.685544344" watchObservedRunningTime="2026-07-06 19:38:36.03388086 +0000 UTC m=+673.687448460" Jul 06 19:38:43.009469 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:43.009422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.611842 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:44.611806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:44.612214 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:44.611890 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:44.624114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:44.624093 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:45.070119 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:45.070092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w" Jul 06 19:38:56.134874 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:38:56.134838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.297102 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.297060 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q"] Jul 06 19:39:00.300138 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.300119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.303368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.303351 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:39:00.310515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.310492 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q"] Jul 06 19:39:00.326377 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.326352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.381310 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9sgg\" (UniqueName: \"kubernetes.io/projected/88f7b67e-dd33-4386-8deb-1a93456194b5-kube-api-access-m9sgg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.381422 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.381422 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f7b67e-dd33-4386-8deb-1a93456194b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.381422 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.381532 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.381532 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.381515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482561 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f7b67e-dd33-4386-8deb-1a93456194b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482678 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482794 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.482794 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9sgg\" (UniqueName: \"kubernetes.io/projected/88f7b67e-dd33-4386-8deb-1a93456194b5-kube-api-access-m9sgg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.483001 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.482980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.483111 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.483089 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.483210 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.483127 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.484882 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.484856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f7b67e-dd33-4386-8deb-1a93456194b5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.485135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.485117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f7b67e-dd33-4386-8deb-1a93456194b5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.490932 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.490912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9sgg\" (UniqueName: \"kubernetes.io/projected/88f7b67e-dd33-4386-8deb-1a93456194b5-kube-api-access-m9sgg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q\" (UID: \"88f7b67e-dd33-4386-8deb-1a93456194b5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.610749 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.610692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:00.746821 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:00.746789 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q"] Jul 06 19:39:00.748556 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:39:00.748525 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f7b67e_dd33_4386_8deb_1a93456194b5.slice/crio-1c0f9ebe737e94e4c68a661c1f54fb03cbc8bb19451e2c8b3aeab480678c224f WatchSource:0}: Error finding container 1c0f9ebe737e94e4c68a661c1f54fb03cbc8bb19451e2c8b3aeab480678c224f: Status 404 returned error can't find the container with id 1c0f9ebe737e94e4c68a661c1f54fb03cbc8bb19451e2c8b3aeab480678c224f Jul 06 19:39:01.140351 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:01.140309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" event={"ID":"88f7b67e-dd33-4386-8deb-1a93456194b5","Type":"ContainerStarted","Data":"b92bae5fd13c6f0926fcf3fb7096b662904e09159d63cb0180804c29ceb5ef80"} Jul 06 19:39:01.140528 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:01.140373 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" event={"ID":"88f7b67e-dd33-4386-8deb-1a93456194b5","Type":"ContainerStarted","Data":"1c0f9ebe737e94e4c68a661c1f54fb03cbc8bb19451e2c8b3aeab480678c224f"} Jul 06 19:39:01.161035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:01.160992 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" podStartSLOduration=0.984553879 podStartE2EDuration="1.16097694s" totalImagesPullingTime="176.423061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:01.158205442 +0000 UTC m=+698.811773042" watchObservedRunningTime="2026-07-06 19:39:01.16097694 +0000 UTC m=+698.814544539" Jul 06 19:39:10.610888 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:10.610854 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:10.611321 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:10.610951 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:10.622967 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:10.622940 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:11.204825 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:11.204792 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q" Jul 06 19:39:20.891909 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:20.891818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:21.834643 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:21.834608 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:39.563302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:39.563258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:51.982504 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:39:51.982468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:55.676830 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:40:55.676724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:55.845871 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:40:55.845842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:56.926190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:40:56.926147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:57.027403 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:40:57.027374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:22.938043 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:41:22.938006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:23.038981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:41:23.038945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:32.652677 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:41:32.652642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:59.161385 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:41:59.161356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:59.262160 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:41:59.262121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:14.064290 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.064248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:14.163864 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.163834 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:42:14.164072 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.164045 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" podUID="908560d8-23e5-43be-a386-a3b3e99824d6" containerName="manager" containerID="cri-o://c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff" gracePeriod=30 Jul 06 19:42:14.419922 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.419876 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:42:14.539503 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.539470 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq7n9\" (UniqueName: \"kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9\") pod \"908560d8-23e5-43be-a386-a3b3e99824d6\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " Jul 06 19:42:14.539672 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.539553 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert\") pod \"908560d8-23e5-43be-a386-a3b3e99824d6\" (UID: \"908560d8-23e5-43be-a386-a3b3e99824d6\") " Jul 06 19:42:14.541534 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.541501 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "908560d8-23e5-43be-a386-a3b3e99824d6" (UID: "908560d8-23e5-43be-a386-a3b3e99824d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:42:14.541800 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.541779 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9" (OuterVolumeSpecName: "kube-api-access-rq7n9") pod "908560d8-23e5-43be-a386-a3b3e99824d6" (UID: "908560d8-23e5-43be-a386-a3b3e99824d6"). InnerVolumeSpecName "kube-api-access-rq7n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:42:14.640947 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.640878 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/908560d8-23e5-43be-a386-a3b3e99824d6-webhook-cert\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:42:14.640947 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:14.640921 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7n9\" (UniqueName: \"kubernetes.io/projected/908560d8-23e5-43be-a386-a3b3e99824d6-kube-api-access-rq7n9\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:42:15.163368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.163332 2577 generic.go:356] "Generic (PLEG): container finished" podID="908560d8-23e5-43be-a386-a3b3e99824d6" containerID="c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff" exitCode=0 Jul 06 19:42:15.163854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.163385 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" Jul 06 19:42:15.163854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.163414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" event={"ID":"908560d8-23e5-43be-a386-a3b3e99824d6","Type":"ContainerDied","Data":"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff"} Jul 06 19:42:15.163854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.163473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-7wkvh" event={"ID":"908560d8-23e5-43be-a386-a3b3e99824d6","Type":"ContainerDied","Data":"8f7b6d3bdfce0bd68a8080df7d6c358e81e34fae9f37261269f9e85efc3a0d8e"} Jul 06 19:42:15.163854 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.163502 2577 scope.go:117] "RemoveContainer" containerID="c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff" Jul 06 19:42:15.172679 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.172661 2577 scope.go:117] "RemoveContainer" containerID="c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff" Jul 06 19:42:15.172945 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:42:15.172924 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff\": container with ID starting with c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff not found: ID does not exist" containerID="c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff" Jul 06 19:42:15.173015 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.172953 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff"} err="failed to get container status \"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff\": rpc error: code = NotFound desc = could not find container \"c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff\": container with ID starting with c72dff5e39fc78f5548e843f70c4a72161308b3d8e7e87055d34510aea3a5dff not found: ID does not exist" Jul 06 19:42:15.185889 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.185859 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:42:15.190471 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.190448 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-7wkvh"] Jul 06 19:42:15.886197 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:15.886160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:16.023730 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.023693 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b8f5864f-n29mx"] Jul 06 19:42:16.024148 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.024131 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908560d8-23e5-43be-a386-a3b3e99824d6" containerName="manager" Jul 06 19:42:16.024254 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.024151 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="908560d8-23e5-43be-a386-a3b3e99824d6" containerName="manager" Jul 06 19:42:16.024309 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.024268 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="908560d8-23e5-43be-a386-a3b3e99824d6" containerName="manager" Jul 06 19:42:16.028673 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.028653 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.034127 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.034108 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-86whh\"" Jul 06 19:42:16.034251 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.034164 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:42:16.034803 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.034784 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:42:16.039257 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.039237 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-n29mx"] Jul 06 19:42:16.154262 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.154183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctjt\" (UniqueName: \"kubernetes.io/projected/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-kube-api-access-9ctjt\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.154262 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.154233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-webhook-cert\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.254921 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.254866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-webhook-cert\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.255368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.254997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctjt\" (UniqueName: \"kubernetes.io/projected/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-kube-api-access-9ctjt\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.257339 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.257315 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-webhook-cert\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.264089 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.264063 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctjt\" (UniqueName: \"kubernetes.io/projected/7e12732d-bc55-4abb-a28b-f2b5bdbc4188-kube-api-access-9ctjt\") pod \"maas-controller-84b8f5864f-n29mx\" (UID: \"7e12732d-bc55-4abb-a28b-f2b5bdbc4188\") " pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.339715 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.339687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:16.475728 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.475702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b8f5864f-n29mx"] Jul 06 19:42:16.477566 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:42:16.477538 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e12732d_bc55_4abb_a28b_f2b5bdbc4188.slice/crio-78bfe90f9eee6e40cd622b094112d0a36e47dacc34c7a9a1bb1cf8b539e5030c WatchSource:0}: Error finding container 78bfe90f9eee6e40cd622b094112d0a36e47dacc34c7a9a1bb1cf8b539e5030c: Status 404 returned error can't find the container with id 78bfe90f9eee6e40cd622b094112d0a36e47dacc34c7a9a1bb1cf8b539e5030c Jul 06 19:42:16.809297 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:16.809268 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908560d8-23e5-43be-a386-a3b3e99824d6" path="/var/lib/kubelet/pods/908560d8-23e5-43be-a386-a3b3e99824d6/volumes" Jul 06 19:42:17.178580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:17.178496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-n29mx" event={"ID":"7e12732d-bc55-4abb-a28b-f2b5bdbc4188","Type":"ContainerStarted","Data":"ee73137859e25088ee7d40e785e77e78668b629e228492d95ac787c5aa16b379"} Jul 06 19:42:17.178580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:17.178543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b8f5864f-n29mx" event={"ID":"7e12732d-bc55-4abb-a28b-f2b5bdbc4188","Type":"ContainerStarted","Data":"78bfe90f9eee6e40cd622b094112d0a36e47dacc34c7a9a1bb1cf8b539e5030c"} Jul 06 19:42:17.178580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:17.178572 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:17.202384 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:17.202330 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b8f5864f-n29mx" podStartSLOduration=1.706120006 podStartE2EDuration="2.202314186s" totalImagesPullingTime="496.19418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:17.198450671 +0000 UTC m=+894.852018282" watchObservedRunningTime="2026-07-06 19:42:17.202314186 +0000 UTC m=+894.855881832" Jul 06 19:42:28.190201 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:28.190163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b8f5864f-n29mx" Jul 06 19:42:59.681877 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:42:59.681846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:00.834994 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:00.834959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:06.038145 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:06.038073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:06.110567 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:06.110533 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:06.201893 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:06.201839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:06.289992 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:06.289908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:06.407483 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:06.407455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:12.403536 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:12.403498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:12.503781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:43:12.503751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:01.613567 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:44:01.613534 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:01.674949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:44:01.674918 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:00.142000 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.141965 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-dpl42"] Jul 06 19:45:00.145478 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.145458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:00.148740 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.148718 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:45:00.160503 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.160480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8w4\" (UniqueName: \"kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4\") pod \"maas-api-key-cleanup-29722785-dpl42\" (UID: \"77083555-2da8-439a-b958-a1233b7c0975\") " pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:00.160613 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.160548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-dpl42"] Jul 06 19:45:00.261519 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.261483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8w4\" (UniqueName: \"kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4\") pod \"maas-api-key-cleanup-29722785-dpl42\" (UID: \"77083555-2da8-439a-b958-a1233b7c0975\") " pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:00.270839 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.270805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8w4\" (UniqueName: \"kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4\") pod \"maas-api-key-cleanup-29722785-dpl42\" (UID: \"77083555-2da8-439a-b958-a1233b7c0975\") " pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:00.455809 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.455743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:00.595497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.595465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-dpl42"] Jul 06 19:45:00.597243 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:45:00.597210 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77083555_2da8_439a_b958_a1233b7c0975.slice/crio-738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135 WatchSource:0}: Error finding container 738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135: Status 404 returned error can't find the container with id 738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135 Jul 06 19:45:00.599370 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:00.599347 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:45:01.037007 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:01.036968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" event={"ID":"77083555-2da8-439a-b958-a1233b7c0975","Type":"ContainerStarted","Data":"738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135"} Jul 06 19:45:04.056725 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:04.056693 2577 generic.go:356] "Generic (PLEG): container finished" podID="77083555-2da8-439a-b958-a1233b7c0975" containerID="08cb12055b8ee351af1d67998738ac057d2c67697a2f09559c129d1172549ce0" exitCode=0 Jul 06 19:45:04.057106 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:04.056735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" event={"ID":"77083555-2da8-439a-b958-a1233b7c0975","Type":"ContainerDied","Data":"08cb12055b8ee351af1d67998738ac057d2c67697a2f09559c129d1172549ce0"} Jul 06 19:45:05.193949 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:05.193926 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:05.306799 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:05.306770 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8w4\" (UniqueName: \"kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4\") pod \"77083555-2da8-439a-b958-a1233b7c0975\" (UID: \"77083555-2da8-439a-b958-a1233b7c0975\") " Jul 06 19:45:05.308693 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:05.308668 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4" (OuterVolumeSpecName: "kube-api-access-zn8w4") pod "77083555-2da8-439a-b958-a1233b7c0975" (UID: "77083555-2da8-439a-b958-a1233b7c0975"). InnerVolumeSpecName "kube-api-access-zn8w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:05.408240 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:05.408214 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8w4\" (UniqueName: \"kubernetes.io/projected/77083555-2da8-439a-b958-a1233b7c0975-kube-api-access-zn8w4\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:45:06.069631 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:06.069602 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" Jul 06 19:45:06.069826 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:06.069649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-dpl42" event={"ID":"77083555-2da8-439a-b958-a1233b7c0975","Type":"ContainerDied","Data":"738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135"} Jul 06 19:45:06.069826 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:06.069693 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738ba7ae6031e262ab503ece51989f6d564348f905c1d2f8f5f46b90d04e9135" Jul 06 19:45:55.381675 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:45:55.381583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:47:06.460642 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:06.460604 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:47:06.461170 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:06.460810 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" podUID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" containerName="manager" containerID="cri-o://b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07" gracePeriod=10 Jul 06 19:47:09.322756 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.322731 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:47:09.475828 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.475805 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume\") pod \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " Jul 06 19:47:09.475993 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.475869 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fff6h\" (UniqueName: \"kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h\") pod \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\" (UID: \"ae9458a0-518c-4292-bf9f-d6ed6937ce3c\") " Jul 06 19:47:09.476174 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.476152 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ae9458a0-518c-4292-bf9f-d6ed6937ce3c" (UID: "ae9458a0-518c-4292-bf9f-d6ed6937ce3c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:09.477861 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.477828 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h" (OuterVolumeSpecName: "kube-api-access-fff6h") pod "ae9458a0-518c-4292-bf9f-d6ed6937ce3c" (UID: "ae9458a0-518c-4292-bf9f-d6ed6937ce3c"). InnerVolumeSpecName "kube-api-access-fff6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:47:09.576790 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.576762 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-extensions-socket-volume\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:47:09.576790 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.576785 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fff6h\" (UniqueName: \"kubernetes.io/projected/ae9458a0-518c-4292-bf9f-d6ed6937ce3c-kube-api-access-fff6h\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:47:09.723599 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.723573 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" containerID="b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07" exitCode=0 Jul 06 19:47:09.723750 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.723610 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" Jul 06 19:47:09.723750 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.723622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" event={"ID":"ae9458a0-518c-4292-bf9f-d6ed6937ce3c","Type":"ContainerDied","Data":"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07"} Jul 06 19:47:09.723750 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.723677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r" event={"ID":"ae9458a0-518c-4292-bf9f-d6ed6937ce3c","Type":"ContainerDied","Data":"21b1e53cea20c2e5ae05aa239283c9a0c77ff9db1ea83f58fd74cc5fc119cdb2"} Jul 06 19:47:09.723750 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.723700 2577 scope.go:117] "RemoveContainer" containerID="b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07" Jul 06 19:47:09.734141 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.734110 2577 scope.go:117] "RemoveContainer" containerID="b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07" Jul 06 19:47:09.734431 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:47:09.734414 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07\": container with ID starting with b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07 not found: ID does not exist" containerID="b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07" Jul 06 19:47:09.734496 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.734440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07"} err="failed to get container status \"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07\": rpc error: code = NotFound desc = could not find container \"b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07\": container with ID starting with b49cfa5240b537877c31dd70c188c9f27d590231c2277d0546dd485cd2171d07 not found: ID does not exist" Jul 06 19:47:09.747781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.747760 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:47:09.751704 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:09.751686 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dfx2r"] Jul 06 19:47:10.809184 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:47:10.809150 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" path="/var/lib/kubelet/pods/ae9458a0-518c-4292-bf9f-d6ed6937ce3c/volumes" Jul 06 19:48:07.732063 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732028 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2"] Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732372 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77083555-2da8-439a-b958-a1233b7c0975" containerName="cleanup" Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732383 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77083555-2da8-439a-b958-a1233b7c0975" containerName="cleanup" Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732402 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" containerName="manager" Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732407 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" containerName="manager" Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732461 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9458a0-518c-4292-bf9f-d6ed6937ce3c" containerName="manager" Jul 06 19:48:07.734468 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.732473 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77083555-2da8-439a-b958-a1233b7c0975" containerName="cleanup" Jul 06 19:48:07.735418 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.735402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.738669 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.738649 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z899f\"" Jul 06 19:48:07.752044 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.752021 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2"] Jul 06 19:48:07.820511 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.820485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgg9\" (UniqueName: \"kubernetes.io/projected/7fea671f-f8a2-4240-a917-9332ce208f4a-kube-api-access-cbgg9\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.820627 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.820541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fea671f-f8a2-4240-a917-9332ce208f4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.921238 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.921213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fea671f-f8a2-4240-a917-9332ce208f4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.921358 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.921270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgg9\" (UniqueName: \"kubernetes.io/projected/7fea671f-f8a2-4240-a917-9332ce208f4a-kube-api-access-cbgg9\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.921635 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.921618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fea671f-f8a2-4240-a917-9332ce208f4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:07.931311 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:07.931288 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgg9\" (UniqueName: \"kubernetes.io/projected/7fea671f-f8a2-4240-a917-9332ce208f4a-kube-api-access-cbgg9\") pod \"kuadrant-operator-controller-manager-86878c769f-fw6w2\" (UID: \"7fea671f-f8a2-4240-a917-9332ce208f4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:08.045368 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:08.045288 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:08.190748 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:08.190722 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2"] Jul 06 19:48:08.192413 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:48:08.192389 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fea671f_f8a2_4240_a917_9332ce208f4a.slice/crio-6098b6b8a0489ace4b3e28342c45912f6dcca831d7cfcdb113aa579e147a1414 WatchSource:0}: Error finding container 6098b6b8a0489ace4b3e28342c45912f6dcca831d7cfcdb113aa579e147a1414: Status 404 returned error can't find the container with id 6098b6b8a0489ace4b3e28342c45912f6dcca831d7cfcdb113aa579e147a1414 Jul 06 19:48:09.027523 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:09.027486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" event={"ID":"7fea671f-f8a2-4240-a917-9332ce208f4a","Type":"ContainerStarted","Data":"aa443665a1f83ef2c749feb7289b58e2d0042349235e6dbb3de35bad4826e749"} Jul 06 19:48:09.027930 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:09.027545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" event={"ID":"7fea671f-f8a2-4240-a917-9332ce208f4a","Type":"ContainerStarted","Data":"6098b6b8a0489ace4b3e28342c45912f6dcca831d7cfcdb113aa579e147a1414"} Jul 06 19:48:09.027930 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:09.027584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:09.056413 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:09.056361 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" podStartSLOduration=2.0563467 podStartE2EDuration="2.0563467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:48:09.053337094 +0000 UTC m=+1246.706904694" watchObservedRunningTime="2026-07-06 19:48:09.0563467 +0000 UTC m=+1246.709914298" Jul 06 19:48:20.035006 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:20.034976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fw6w2" Jul 06 19:48:21.880753 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:48:21.880699 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:27.436547 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:49:27.436512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:54.829393 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:53:54.829362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:59.410216 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:53:59.410176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:07.694004 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:07.693959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:12.819552 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:12.819520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:15.731374 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:15.731284 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:15.817840 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:15.817809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:16.234464 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:16.234431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:22.139757 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:22.139719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:22.213976 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:22.213946 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:22.412125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:22.412032 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:22.503497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:22.503449 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:23.715370 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.715339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:23.847064 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.847033 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:23.850332 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.850316 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:23.854528 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.854497 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:54:23.855240 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.855221 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8f9e41e4-serving-cert\"" Jul 06 19:54:23.887509 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.887476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:23.888133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.888113 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:23.981478 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.981448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76cq\" (UniqueName: \"kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:23.981611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.981494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:23.981611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:23.981531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.039461 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.039434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:24.082286 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.082265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.082449 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.082431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76cq\" (UniqueName: \"kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.082497 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.082477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.082978 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.082961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.084838 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.084821 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.101559 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.101533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76cq\" (UniqueName: \"kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq\") pod \"maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.144611 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.144570 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:24.161191 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.161170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.298279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.298255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:24.299329 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:54:24.299302 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8be438_26c7_48cb_9679_e70a30f03246.slice/crio-24fd25183dc92ffba97e325a56c3d5828695a47cf9863059a4a19d57d9655709 WatchSource:0}: Error finding container 24fd25183dc92ffba97e325a56c3d5828695a47cf9863059a4a19d57d9655709: Status 404 returned error can't find the container with id 24fd25183dc92ffba97e325a56c3d5828695a47cf9863059a4a19d57d9655709 Jul 06 19:54:24.300585 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.300565 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:54:24.540754 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.540632 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:24.635442 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.635416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:24.997788 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.997750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" event={"ID":"cc8be438-26c7-48cb-9679-e70a30f03246","Type":"ContainerStarted","Data":"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091"} Jul 06 19:54:24.998261 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.997926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:24.998261 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:24.997968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" event={"ID":"cc8be438-26c7-48cb-9679-e70a30f03246","Type":"ContainerStarted","Data":"24fd25183dc92ffba97e325a56c3d5828695a47cf9863059a4a19d57d9655709"} Jul 06 19:54:25.017263 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:25.017218 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" podStartSLOduration=1.5215023479999998 podStartE2EDuration="2.017205177s" totalImagesPullingTime="495.702829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:25.015169827 +0000 UTC m=+1622.668737427" watchObservedRunningTime="2026-07-06 19:54:25.017205177 +0000 UTC m=+1622.670773132" Jul 06 19:54:25.994870 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:25.994839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:28.822439 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:28.822405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:28.979912 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:28.979873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:29.076064 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.075973 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:29.212185 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.212155 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:29.212375 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.212348 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" podUID="cc8be438-26c7-48cb-9679-e70a30f03246" containerName="maas-api" containerID="cri-o://6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091" gracePeriod=30 Jul 06 19:54:29.218530 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.218508 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:29.246412 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.246391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:29.463604 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.463580 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:29.629375 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.629303 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p76cq\" (UniqueName: \"kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq\") pod \"cc8be438-26c7-48cb-9679-e70a30f03246\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " Jul 06 19:54:29.629375 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.629349 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca\") pod \"cc8be438-26c7-48cb-9679-e70a30f03246\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " Jul 06 19:54:29.629565 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.629386 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls\") pod \"cc8be438-26c7-48cb-9679-e70a30f03246\" (UID: \"cc8be438-26c7-48cb-9679-e70a30f03246\") " Jul 06 19:54:29.629765 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.629739 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc8be438-26c7-48cb-9679-e70a30f03246" (UID: "cc8be438-26c7-48cb-9679-e70a30f03246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:54:29.631405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.631368 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc8be438-26c7-48cb-9679-e70a30f03246" (UID: "cc8be438-26c7-48cb-9679-e70a30f03246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:29.631405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.631391 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq" (OuterVolumeSpecName: "kube-api-access-p76cq") pod "cc8be438-26c7-48cb-9679-e70a30f03246" (UID: "cc8be438-26c7-48cb-9679-e70a30f03246"). InnerVolumeSpecName "kube-api-access-p76cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:29.730162 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.730133 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76cq\" (UniqueName: \"kubernetes.io/projected/cc8be438-26c7-48cb-9679-e70a30f03246-kube-api-access-p76cq\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:29.730162 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.730157 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc8be438-26c7-48cb-9679-e70a30f03246-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:29.730162 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:29.730166 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc8be438-26c7-48cb-9679-e70a30f03246-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:30.025546 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.025507 2577 generic.go:356] "Generic (PLEG): container finished" podID="cc8be438-26c7-48cb-9679-e70a30f03246" containerID="6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091" exitCode=0 Jul 06 19:54:30.026005 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.025556 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" Jul 06 19:54:30.026005 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.025559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" event={"ID":"cc8be438-26c7-48cb-9679-e70a30f03246","Type":"ContainerDied","Data":"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091"} Jul 06 19:54:30.026005 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.025612 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d" event={"ID":"cc8be438-26c7-48cb-9679-e70a30f03246","Type":"ContainerDied","Data":"24fd25183dc92ffba97e325a56c3d5828695a47cf9863059a4a19d57d9655709"} Jul 06 19:54:30.026005 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.025646 2577 scope.go:117] "RemoveContainer" containerID="6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091" Jul 06 19:54:30.038119 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.038094 2577 scope.go:117] "RemoveContainer" containerID="6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091" Jul 06 19:54:30.038523 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:54:30.038499 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091\": container with ID starting with 6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091 not found: ID does not exist" containerID="6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091" Jul 06 19:54:30.038614 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.038537 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091"} err="failed to get container status \"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091\": rpc error: code = NotFound desc = could not find container \"6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091\": container with ID starting with 6aacab729e9e283e5136bf588cd029e167f4da7f50bb8403648506dbbf3b8091 not found: ID does not exist" Jul 06 19:54:30.053003 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.052982 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:30.055520 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.055500 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8f9e41e4-6bddb94b46-mqg7d"] Jul 06 19:54:30.367034 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.367005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:30.810206 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:30.810169 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8be438-26c7-48cb-9679-e70a30f03246" path="/var/lib/kubelet/pods/cc8be438-26c7-48cb-9679-e70a30f03246/volumes" Jul 06 19:54:34.993802 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:34.993771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.063039 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:35.063013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.246335 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:35.246257 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.363070 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:35.363035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.334211 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.334162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.404973 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.404939 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:36.405521 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.405501 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8be438-26c7-48cb-9679-e70a30f03246" containerName="maas-api" Jul 06 19:54:36.405598 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.405524 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8be438-26c7-48cb-9679-e70a30f03246" containerName="maas-api" Jul 06 19:54:36.405663 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.405625 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8be438-26c7-48cb-9679-e70a30f03246" containerName="maas-api" Jul 06 19:54:36.410090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.410068 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.413222 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.413184 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-363e7ca9-serving-cert\"" Jul 06 19:54:36.413445 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.413290 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:54:36.421969 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.421946 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:36.431863 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.431833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.511068 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.511037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.586269 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.586181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.586269 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.586226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddr72\" (UniqueName: \"kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.586472 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.586297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.672957 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.672927 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.687354 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.687331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.687449 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.687409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.687449 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.687431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddr72\" (UniqueName: \"kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.688063 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.688045 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.689727 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.689710 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.697400 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.697371 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddr72\" (UniqueName: \"kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72\") pod \"maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.721411 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.721390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:36.858465 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.858436 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:36.860193 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:54:36.860166 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4b9cd5_61a1_4a28_8568_4563ff0c0382.slice/crio-65683475f07a44fd0ba63ae685832b9ad5030ab9fb4a828481bded15470b5ed3 WatchSource:0}: Error finding container 65683475f07a44fd0ba63ae685832b9ad5030ab9fb4a828481bded15470b5ed3: Status 404 returned error can't find the container with id 65683475f07a44fd0ba63ae685832b9ad5030ab9fb4a828481bded15470b5ed3 Jul 06 19:54:36.995024 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:36.994996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:37.063516 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:37.063479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" event={"ID":"0c4b9cd5-61a1-4a28-8568-4563ff0c0382","Type":"ContainerStarted","Data":"65683475f07a44fd0ba63ae685832b9ad5030ab9fb4a828481bded15470b5ed3"} Jul 06 19:54:37.096361 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:37.096329 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:38.070008 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:38.069972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" event={"ID":"0c4b9cd5-61a1-4a28-8568-4563ff0c0382","Type":"ContainerStarted","Data":"a5e8088ad0c82fc90dacbd0af5a5b38345dd8ddc83096721f79e0742e5442783"} Jul 06 19:54:38.070388 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:38.070095 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:38.088735 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:38.088685 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" podStartSLOduration=1.731701473 podStartE2EDuration="2.088670971s" totalImagesPullingTime="356.969498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:38.08796166 +0000 UTC m=+1635.741529259" watchObservedRunningTime="2026-07-06 19:54:38.088670971 +0000 UTC m=+1635.742238624" Jul 06 19:54:38.864613 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:38.864586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:41.496711 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:41.496681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:41.889116 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:41.889038 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:41.889383 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:41.889282 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" podUID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" containerName="maas-api" containerID="cri-o://a5e8088ad0c82fc90dacbd0af5a5b38345dd8ddc83096721f79e0742e5442783" gracePeriod=30 Jul 06 19:54:41.914699 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:41.914674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:42.092795 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.092767 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" containerID="a5e8088ad0c82fc90dacbd0af5a5b38345dd8ddc83096721f79e0742e5442783" exitCode=0 Jul 06 19:54:42.092959 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.092843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" event={"ID":"0c4b9cd5-61a1-4a28-8568-4563ff0c0382","Type":"ContainerDied","Data":"a5e8088ad0c82fc90dacbd0af5a5b38345dd8ddc83096721f79e0742e5442783"} Jul 06 19:54:42.139823 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.139774 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:42.233341 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.233313 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls\") pod \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " Jul 06 19:54:42.233512 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.233357 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca\") pod \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " Jul 06 19:54:42.233512 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.233381 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddr72\" (UniqueName: \"kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72\") pod \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\" (UID: \"0c4b9cd5-61a1-4a28-8568-4563ff0c0382\") " Jul 06 19:54:42.233715 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.233691 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c4b9cd5-61a1-4a28-8568-4563ff0c0382" (UID: "0c4b9cd5-61a1-4a28-8568-4563ff0c0382"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:54:42.235361 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.235336 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c4b9cd5-61a1-4a28-8568-4563ff0c0382" (UID: "0c4b9cd5-61a1-4a28-8568-4563ff0c0382"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:42.235464 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.235445 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72" (OuterVolumeSpecName: "kube-api-access-ddr72") pod "0c4b9cd5-61a1-4a28-8568-4563ff0c0382" (UID: "0c4b9cd5-61a1-4a28-8568-4563ff0c0382"). InnerVolumeSpecName "kube-api-access-ddr72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:42.334526 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.334498 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.334526 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.334523 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:42.334687 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:42.334534 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddr72\" (UniqueName: \"kubernetes.io/projected/0c4b9cd5-61a1-4a28-8568-4563ff0c0382-kube-api-access-ddr72\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.099221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:43.099149 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" Jul 06 19:54:43.099608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:43.099146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968" event={"ID":"0c4b9cd5-61a1-4a28-8568-4563ff0c0382","Type":"ContainerDied","Data":"65683475f07a44fd0ba63ae685832b9ad5030ab9fb4a828481bded15470b5ed3"} Jul 06 19:54:43.099608 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:43.099277 2577 scope.go:117] "RemoveContainer" containerID="a5e8088ad0c82fc90dacbd0af5a5b38345dd8ddc83096721f79e0742e5442783" Jul 06 19:54:43.118060 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:43.118036 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:43.120729 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:43.120707 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-363e7ca9-56b99c6c7f-hn968"] Jul 06 19:54:44.810409 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:44.810380 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" path="/var/lib/kubelet/pods/0c4b9cd5-61a1-4a28-8568-4563ff0c0382/volumes" Jul 06 19:54:47.662353 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:47.662295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:47.761943 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:47.761890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:49.081124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:49.081091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:53.741093 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:53.741057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:53.830586 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:53.830558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:53.990484 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:53.990454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:54.084092 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:54.084016 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.101666 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.101638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.168653 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.168627 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:54:55.169037 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.169012 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" containerName="maas-api" Jul 06 19:54:55.169037 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.169030 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" containerName="maas-api" Jul 06 19:54:55.169222 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.169121 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4b9cd5-61a1-4a28-8568-4563ff0c0382" containerName="maas-api" Jul 06 19:54:55.172387 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.172365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.175638 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.175616 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:54:55.175889 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.175868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-e1149fa3-serving-cert\"" Jul 06 19:54:55.187507 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.187483 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:54:55.188529 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.188509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.240439 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.240413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.240566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.240445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.240566 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.240492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxmd\" (UniqueName: \"kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.267891 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.267866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.341592 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.341561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxmd\" (UniqueName: \"kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.341733 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.341664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.341733 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.341692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.341894 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:54:55.341862 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-e1149fa3-serving-cert: secret "maas-api-e2e-derive-e1149fa3-serving-cert" not found Jul 06 19:54:55.342011 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:54:55.341988 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls podName:74dde432-d1c7-45b1-8454-7fbc372986d9 nodeName:}" failed. No retries permitted until 2026-07-06 19:54:55.841967292 +0000 UTC m=+1653.495534874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls") pod "maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" (UID: "74dde432-d1c7-45b1-8454-7fbc372986d9") : secret "maas-api-e2e-derive-e1149fa3-serving-cert" not found Jul 06 19:54:55.342598 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.342571 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.353413 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.353360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxmd\" (UniqueName: \"kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.559410 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.559365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.667385 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.667309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.767396 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.767362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:55.846221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.846194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:55.848470 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:55.848449 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") pod \"maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:56.085475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:56.085436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:56.230028 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:56.230000 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:54:56.230891 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:54:56.230862 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dde432_d1c7_45b1_8454_7fbc372986d9.slice/crio-f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5 WatchSource:0}: Error finding container f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5: Status 404 returned error can't find the container with id f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5 Jul 06 19:54:57.179582 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:57.179541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" event={"ID":"74dde432-d1c7-45b1-8454-7fbc372986d9","Type":"ContainerStarted","Data":"145bb7b1a2f06ff59825d58a209ae00ffea683cfb4bd4cea4a4a0c332b6f2924"} Jul 06 19:54:57.179820 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:57.179606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" event={"ID":"74dde432-d1c7-45b1-8454-7fbc372986d9","Type":"ContainerStarted","Data":"f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5"} Jul 06 19:54:57.179820 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:57.179761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:54:57.201448 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:57.201405 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" podStartSLOduration=1.824526879 podStartE2EDuration="2.201389972s" totalImagesPullingTime="376.863093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:57.198275387 +0000 UTC m=+1654.851842987" watchObservedRunningTime="2026-07-06 19:54:57.201389972 +0000 UTC m=+1654.854957570" Jul 06 19:54:57.229786 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:57.229759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:59.556767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.556736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:59.768339 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.768307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:59.867953 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.867853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:59.942647 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.942613 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:54:59.942929 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.942880 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" podUID="74dde432-d1c7-45b1-8454-7fbc372986d9" containerName="maas-api" containerID="cri-o://145bb7b1a2f06ff59825d58a209ae00ffea683cfb4bd4cea4a4a0c332b6f2924" gracePeriod=30 Jul 06 19:54:59.972399 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:54:59.972374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:00.197198 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.197173 2577 generic.go:356] "Generic (PLEG): container finished" podID="74dde432-d1c7-45b1-8454-7fbc372986d9" containerID="145bb7b1a2f06ff59825d58a209ae00ffea683cfb4bd4cea4a4a0c332b6f2924" exitCode=0 Jul 06 19:55:00.197302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.197240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" event={"ID":"74dde432-d1c7-45b1-8454-7fbc372986d9","Type":"ContainerDied","Data":"145bb7b1a2f06ff59825d58a209ae00ffea683cfb4bd4cea4a4a0c332b6f2924"} Jul 06 19:55:00.197302 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.197288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" event={"ID":"74dde432-d1c7-45b1-8454-7fbc372986d9","Type":"ContainerDied","Data":"f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5"} Jul 06 19:55:00.197400 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.197306 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b59c7eecbeb356848d82d7bb99cf423cd4e5fe357b28869a24c33685cbb8e5" Jul 06 19:55:00.201261 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.201244 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:55:00.282759 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.282729 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwxmd\" (UniqueName: \"kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd\") pod \"74dde432-d1c7-45b1-8454-7fbc372986d9\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " Jul 06 19:55:00.282930 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.282774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") pod \"74dde432-d1c7-45b1-8454-7fbc372986d9\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " Jul 06 19:55:00.282930 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.282797 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca\") pod \"74dde432-d1c7-45b1-8454-7fbc372986d9\" (UID: \"74dde432-d1c7-45b1-8454-7fbc372986d9\") " Jul 06 19:55:00.283190 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.283169 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74dde432-d1c7-45b1-8454-7fbc372986d9" (UID: "74dde432-d1c7-45b1-8454-7fbc372986d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:55:00.284781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.284763 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74dde432-d1c7-45b1-8454-7fbc372986d9" (UID: "74dde432-d1c7-45b1-8454-7fbc372986d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:55:00.284937 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.284891 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd" (OuterVolumeSpecName: "kube-api-access-gwxmd") pod "74dde432-d1c7-45b1-8454-7fbc372986d9" (UID: "74dde432-d1c7-45b1-8454-7fbc372986d9"). InnerVolumeSpecName "kube-api-access-gwxmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:00.383492 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.383464 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxmd\" (UniqueName: \"kubernetes.io/projected/74dde432-d1c7-45b1-8454-7fbc372986d9-kube-api-access-gwxmd\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:55:00.383492 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.383491 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74dde432-d1c7-45b1-8454-7fbc372986d9-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:55:00.383492 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:00.383502 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dde432-d1c7-45b1-8454-7fbc372986d9-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:55:01.203441 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:01.203355 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498" Jul 06 19:55:01.223189 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:01.223161 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:55:01.229135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:01.229112 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-e1149fa3-7f9b5b78f9-j5498"] Jul 06 19:55:01.839499 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:01.839464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:02.809784 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:02.809744 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74dde432-d1c7-45b1-8454-7fbc372986d9" path="/var/lib/kubelet/pods/74dde432-d1c7-45b1-8454-7fbc372986d9/volumes" Jul 06 19:55:05.659182 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:05.659145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:05.736856 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:05.736829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:06.140135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:06.140083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:06.234392 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:06.234354 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:06.329476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:06.329423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:07.711589 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:07.711555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:13.612505 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:13.612469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:13.715360 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:13.715328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:13.814223 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:13.814186 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:13.914554 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:13.914462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:14.370459 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:14.370409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:40.352487 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:40.352435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:46.141661 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:46.141581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:46.210099 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:46.210062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:46.304008 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:46.303982 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:46.408135 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:46.408055 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:50.488447 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:55:50.488418 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:02.270708 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:02.270674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.147053 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:08.147020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.259966 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:08.259929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.421170 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:08.421089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:18.903164 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:18.903109 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:19.736914 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:19.736865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:25.635031 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:25.634996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:25.745242 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:25.745197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:25.836363 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:25.836335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:25.936839 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:25.936772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:26.714968 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:26.714935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:26.812309 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:26.812278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:26.913105 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:26.913074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.544987 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:28.544950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.430357 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:34.430320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.495138 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:34.495111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:35.115134 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:35.115098 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.131743 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:41.131706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.232090 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:41.232060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.446062 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:41.445979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.748783 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:41.748747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.875758 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:41.875698 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.056680 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.056640 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.150419 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.150389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.211195 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.211162 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:45.211703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.211683 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dde432-d1c7-45b1-8454-7fbc372986d9" containerName="maas-api" Jul 06 19:56:45.211703 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.211702 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dde432-d1c7-45b1-8454-7fbc372986d9" containerName="maas-api" Jul 06 19:56:45.211920 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.211800 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dde432-d1c7-45b1-8454-7fbc372986d9" containerName="maas-api" Jul 06 19:56:45.215225 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.215201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.218397 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.218375 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-7fd56226-serving-cert\"" Jul 06 19:56:45.218515 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.218453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:56:45.235053 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.235028 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:45.266127 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.266093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.313981 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.313912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.314136 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.313984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqg2b\" (UniqueName: \"kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.314136 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.314013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.415209 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.415175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqg2b\" (UniqueName: \"kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.415373 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.415222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.415373 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.415276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.415497 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:56:45.415446 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-7fd56226-serving-cert: secret "maas-api-e2e-mt-7fd56226-serving-cert" not found Jul 06 19:56:45.415552 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:56:45.415541 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls podName:9e7afe25-063c-4fa6-8913-d496687d0b92 nodeName:}" failed. No retries permitted until 2026-07-06 19:56:45.915508173 +0000 UTC m=+1763.569075766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls") pod "maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" (UID: "9e7afe25-063c-4fa6-8913-d496687d0b92") : secret "maas-api-e2e-mt-7fd56226-serving-cert" not found Jul 06 19:56:45.415890 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.415869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.429407 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.429386 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqg2b\" (UniqueName: \"kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.556309 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.556279 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.755951 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.755921 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.855218 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.855190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:45.919511 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.919484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:45.921841 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:45.921819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") pod \"maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:46.125976 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:46.125869 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:46.269931 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:46.269887 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:46.270661 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:56:46.270639 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7afe25_063c_4fa6_8913_d496687d0b92.slice/crio-3ddcd92554a3c5163f72c9a27d09793e4422d0931bea1db4d638b2e6b12cfc11 WatchSource:0}: Error finding container 3ddcd92554a3c5163f72c9a27d09793e4422d0931bea1db4d638b2e6b12cfc11: Status 404 returned error can't find the container with id 3ddcd92554a3c5163f72c9a27d09793e4422d0931bea1db4d638b2e6b12cfc11 Jul 06 19:56:46.758207 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:46.758184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" event={"ID":"9e7afe25-063c-4fa6-8913-d496687d0b92","Type":"ContainerStarted","Data":"3ddcd92554a3c5163f72c9a27d09793e4422d0931bea1db4d638b2e6b12cfc11"} Jul 06 19:56:47.770193 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:47.770154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" event={"ID":"9e7afe25-063c-4fa6-8913-d496687d0b92","Type":"ContainerStarted","Data":"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503"} Jul 06 19:56:47.770814 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:47.770791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:47.793232 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:47.793186 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" podStartSLOduration=2.339082754 podStartE2EDuration="2.79317397s" totalImagesPullingTime="454.091216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:47.791354149 +0000 UTC m=+1765.444921750" watchObservedRunningTime="2026-07-06 19:56:47.79317397 +0000 UTC m=+1765.446741568" Jul 06 19:56:49.565636 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:49.565602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:50.229736 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:50.229693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:50.326872 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:50.326842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:50.685781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:50.685685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:53.780580 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:53.780553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:56.616036 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:56.615993 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:56.715169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:56.715137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:57.052594 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.052559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:57.373242 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.373153 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:57.373418 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.373380 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" podUID="9e7afe25-063c-4fa6-8913-d496687d0b92" containerName="maas-api" containerID="cri-o://62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503" gracePeriod=30 Jul 06 19:56:57.398876 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.398847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:57.466982 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.466954 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:57.629380 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.629331 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:57.718004 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.717977 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca\") pod \"9e7afe25-063c-4fa6-8913-d496687d0b92\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " Jul 06 19:56:57.718125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.718017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqg2b\" (UniqueName: \"kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b\") pod \"9e7afe25-063c-4fa6-8913-d496687d0b92\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " Jul 06 19:56:57.718125 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.718037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") pod \"9e7afe25-063c-4fa6-8913-d496687d0b92\" (UID: \"9e7afe25-063c-4fa6-8913-d496687d0b92\") " Jul 06 19:56:57.718393 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.718370 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e7afe25-063c-4fa6-8913-d496687d0b92" (UID: "9e7afe25-063c-4fa6-8913-d496687d0b92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:56:57.720114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.720091 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e7afe25-063c-4fa6-8913-d496687d0b92" (UID: "9e7afe25-063c-4fa6-8913-d496687d0b92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:56:57.720114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.720110 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b" (OuterVolumeSpecName: "kube-api-access-dqg2b") pod "9e7afe25-063c-4fa6-8913-d496687d0b92" (UID: "9e7afe25-063c-4fa6-8913-d496687d0b92"). InnerVolumeSpecName "kube-api-access-dqg2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:57.819077 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.819051 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7afe25-063c-4fa6-8913-d496687d0b92-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:56:57.819077 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.819077 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqg2b\" (UniqueName: \"kubernetes.io/projected/9e7afe25-063c-4fa6-8913-d496687d0b92-kube-api-access-dqg2b\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:56:57.819221 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.819093 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7afe25-063c-4fa6-8913-d496687d0b92-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:56:57.823927 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.823880 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e7afe25-063c-4fa6-8913-d496687d0b92" containerID="62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503" exitCode=0 Jul 06 19:56:57.824035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.823940 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" Jul 06 19:56:57.824035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.823946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" event={"ID":"9e7afe25-063c-4fa6-8913-d496687d0b92","Type":"ContainerDied","Data":"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503"} Jul 06 19:56:57.824035 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.824015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl" event={"ID":"9e7afe25-063c-4fa6-8913-d496687d0b92","Type":"ContainerDied","Data":"3ddcd92554a3c5163f72c9a27d09793e4422d0931bea1db4d638b2e6b12cfc11"} Jul 06 19:56:57.824169 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.824050 2577 scope.go:117] "RemoveContainer" containerID="62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503" Jul 06 19:56:57.833367 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.833351 2577 scope.go:117] "RemoveContainer" containerID="62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503" Jul 06 19:56:57.833609 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:56:57.833584 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503\": container with ID starting with 62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503 not found: ID does not exist" containerID="62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503" Jul 06 19:56:57.833724 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.833617 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503"} err="failed to get container status \"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503\": rpc error: code = NotFound desc = could not find container \"62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503\": container with ID starting with 62e35d1803d1bc3dc1b9bbc87604893d4aadebafd6463cdbf7d9394737637503 not found: ID does not exist" Jul 06 19:56:57.850627 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.850603 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:57.859582 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:57.859563 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-7fd56226-56b45f645f-xjjkl"] Jul 06 19:56:58.816475 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:56:58.816441 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7afe25-063c-4fa6-8913-d496687d0b92" path="/var/lib/kubelet/pods/9e7afe25-063c-4fa6-8913-d496687d0b92/volumes" Jul 06 19:57:03.507338 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:03.507294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:03.613405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:03.613378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:03.814641 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:03.814565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:04.019289 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:04.019252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:04.167573 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:04.167490 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:04.263592 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:04.263546 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:04.377307 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:04.377276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:10.178837 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:10.178799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:10.282945 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:10.282889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:10.399479 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:10.399448 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:10.592347 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:10.592317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:16.732781 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:16.732702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:16.836295 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:16.836269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:23.649574 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:23.649539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:23.976141 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:23.976111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:24.312279 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:24.312202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:27.275835 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.275775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:27.357712 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.357679 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:57:27.358476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.358449 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7afe25-063c-4fa6-8913-d496687d0b92" containerName="maas-api" Jul 06 19:57:27.358476 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.358476 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7afe25-063c-4fa6-8913-d496687d0b92" containerName="maas-api" Jul 06 19:57:27.358671 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.358569 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7afe25-063c-4fa6-8913-d496687d0b92" containerName="maas-api" Jul 06 19:57:27.361623 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.361602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.364667 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.364470 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 19:57:27.364993 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.364973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-f938018c-serving-cert\"" Jul 06 19:57:27.374957 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.374937 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:57:27.392715 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.392692 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:27.465733 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.465705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84wt\" (UniqueName: \"kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.465852 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.465752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.465894 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.465842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.480052 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.480029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:27.567227 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.567157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84wt\" (UniqueName: \"kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.567227 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.567203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.567357 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.567247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.567395 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:57:27.567373 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-f938018c-serving-cert: secret "maas-api-e2e-mt-f938018c-serving-cert" not found Jul 06 19:57:27.567444 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:57:27.567433 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls podName:ff7a342c-a5b9-4831-a994-9844e99e01e2 nodeName:}" failed. No retries permitted until 2026-07-06 19:57:28.067416413 +0000 UTC m=+1805.720983990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls") pod "maas-api-e2e-mt-f938018c-5db89b6897-427vt" (UID: "ff7a342c-a5b9-4831-a994-9844e99e01e2") : secret "maas-api-e2e-mt-f938018c-serving-cert" not found Jul 06 19:57:27.568046 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.568028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.578975 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.578956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84wt\" (UniqueName: \"kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:27.824551 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.824470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:27.981435 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:27.981405 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:28.071539 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.071509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:28.074071 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.074045 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") pod \"maas-api-e2e-mt-f938018c-5db89b6897-427vt\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:28.077865 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.077810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:28.273526 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.273494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:28.424181 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.424153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:57:28.425785 ip-10-0-130-143 kubenswrapper[2577]: W0706 19:57:28.425760 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7a342c_a5b9_4831_a994_9844e99e01e2.slice/crio-d415de0019e7d2f15feb5ff81a685e6a8b80ddd39272d13914b289ee8efc94a6 WatchSource:0}: Error finding container d415de0019e7d2f15feb5ff81a685e6a8b80ddd39272d13914b289ee8efc94a6: Status 404 returned error can't find the container with id d415de0019e7d2f15feb5ff81a685e6a8b80ddd39272d13914b289ee8efc94a6 Jul 06 19:57:28.986248 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.986212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" event={"ID":"ff7a342c-a5b9-4831-a994-9844e99e01e2","Type":"ContainerStarted","Data":"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244"} Jul 06 19:57:28.986405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.986265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" event={"ID":"ff7a342c-a5b9-4831-a994-9844e99e01e2","Type":"ContainerStarted","Data":"d415de0019e7d2f15feb5ff81a685e6a8b80ddd39272d13914b289ee8efc94a6"} Jul 06 19:57:28.986405 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:28.986349 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:29.008303 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:29.008257 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" podStartSLOduration=1.641729988 podStartE2EDuration="2.00824375s" totalImagesPullingTime="366.513762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:57:29.00693346 +0000 UTC m=+1806.660501051" watchObservedRunningTime="2026-07-06 19:57:29.00824375 +0000 UTC m=+1806.661811373" Jul 06 19:57:31.814508 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:31.814473 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:34.996955 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:34.996932 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:57:37.310061 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:37.310028 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:37.410651 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:37.410616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:42.128833 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:42.128797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:48.839668 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:48.839636 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:48.940161 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:48.940131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:59.313263 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:59.313231 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:59.683774 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:59.683696 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:57:59.683953 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:59.683927 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" podUID="ff7a342c-a5b9-4831-a994-9844e99e01e2" containerName="maas-api" containerID="cri-o://a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244" gracePeriod=30 Jul 06 19:57:59.704282 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:59.704254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:59.939133 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:57:59.939081 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:58:00.021695 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.021664 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca\") pod \"ff7a342c-a5b9-4831-a994-9844e99e01e2\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " Jul 06 19:58:00.021862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.021711 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w84wt\" (UniqueName: \"kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt\") pod \"ff7a342c-a5b9-4831-a994-9844e99e01e2\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " Jul 06 19:58:00.021862 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.021766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") pod \"ff7a342c-a5b9-4831-a994-9844e99e01e2\" (UID: \"ff7a342c-a5b9-4831-a994-9844e99e01e2\") " Jul 06 19:58:00.022053 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.022031 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff7a342c-a5b9-4831-a994-9844e99e01e2" (UID: "ff7a342c-a5b9-4831-a994-9844e99e01e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:58:00.023748 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.023726 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff7a342c-a5b9-4831-a994-9844e99e01e2" (UID: "ff7a342c-a5b9-4831-a994-9844e99e01e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:58:00.023850 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.023730 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt" (OuterVolumeSpecName: "kube-api-access-w84wt") pod "ff7a342c-a5b9-4831-a994-9844e99e01e2" (UID: "ff7a342c-a5b9-4831-a994-9844e99e01e2"). InnerVolumeSpecName "kube-api-access-w84wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:00.123237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.123210 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff7a342c-a5b9-4831-a994-9844e99e01e2-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:58:00.123237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.123235 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7a342c-a5b9-4831-a994-9844e99e01e2-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:58:00.123237 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.123245 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84wt\" (UniqueName: \"kubernetes.io/projected/ff7a342c-a5b9-4831-a994-9844e99e01e2-kube-api-access-w84wt\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 19:58:00.153626 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.153594 2577 generic.go:356] "Generic (PLEG): container finished" podID="ff7a342c-a5b9-4831-a994-9844e99e01e2" containerID="a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244" exitCode=0 Jul 06 19:58:00.153767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.153640 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" Jul 06 19:58:00.153767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.153658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" event={"ID":"ff7a342c-a5b9-4831-a994-9844e99e01e2","Type":"ContainerDied","Data":"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244"} Jul 06 19:58:00.153767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.153712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt" event={"ID":"ff7a342c-a5b9-4831-a994-9844e99e01e2","Type":"ContainerDied","Data":"d415de0019e7d2f15feb5ff81a685e6a8b80ddd39272d13914b289ee8efc94a6"} Jul 06 19:58:00.153767 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.153743 2577 scope.go:117] "RemoveContainer" containerID="a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244" Jul 06 19:58:00.163213 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.163196 2577 scope.go:117] "RemoveContainer" containerID="a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244" Jul 06 19:58:00.163472 ip-10-0-130-143 kubenswrapper[2577]: E0706 19:58:00.163451 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244\": container with ID starting with a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244 not found: ID does not exist" containerID="a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244" Jul 06 19:58:00.163523 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.163481 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244"} err="failed to get container status \"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244\": rpc error: code = NotFound desc = could not find container \"a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244\": container with ID starting with a48e7bd830db716ed2510f5ca18c6584c9c0f1f37b89d95b1465cdbc45825244 not found: ID does not exist" Jul 06 19:58:00.176114 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.176093 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:58:00.180132 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.180107 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-f938018c-5db89b6897-427vt"] Jul 06 19:58:00.810684 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:00.810654 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff7a342c-a5b9-4831-a994-9844e99e01e2" path="/var/lib/kubelet/pods/ff7a342c-a5b9-4831-a994-9844e99e01e2/volumes" Jul 06 19:58:05.450794 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:05.450753 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:10.503775 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:10.503736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.243800 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:22.243765 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.299633 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:22.299602 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:32.182263 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:32.182233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:32.514974 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:32.514944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:38.099400 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:38.099360 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:39.511639 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:39.511601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:39.577576 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:58:39.577540 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:01.693454 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:01.693420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:02.729281 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:02.729248 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:03.313124 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:03.313091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:03.413414 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:03.413375 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:03.512855 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:03.512826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:04.342612 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:04.342574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:04.434500 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:04.434465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:04.531838 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:04.531809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:05.925814 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:05.925782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.298213 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:12.298182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:12.358265 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:12.358235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:18.079603 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:18.079569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.194193 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:24.194164 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.266474 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:24.266442 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.384569 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:24.384537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.478065 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:24.478035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:34.976488 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:34.976451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:35.339655 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:35.339572 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:54.390148 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:54.390118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:56.290868 ip-10-0-130-143 kubenswrapper[2577]: I0706 19:59:56.290833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:00.140931 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.140886 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-959fl"] Jul 06 20:00:00.141356 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.141269 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7a342c-a5b9-4831-a994-9844e99e01e2" containerName="maas-api" Jul 06 20:00:00.141356 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.141281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7a342c-a5b9-4831-a994-9844e99e01e2" containerName="maas-api" Jul 06 20:00:00.141356 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.141351 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7a342c-a5b9-4831-a994-9844e99e01e2" containerName="maas-api" Jul 06 20:00:00.144586 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.144571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:00.147545 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.147525 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 20:00:00.153350 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.153324 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-959fl"] Jul 06 20:00:00.198800 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.198775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wr7\" (UniqueName: \"kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7\") pod \"maas-api-key-cleanup-29722800-959fl\" (UID: \"eceb335d-dead-4547-b3b0-4208e662d384\") " pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:00.300144 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.300108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wr7\" (UniqueName: \"kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7\") pod \"maas-api-key-cleanup-29722800-959fl\" (UID: \"eceb335d-dead-4547-b3b0-4208e662d384\") " pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:00.309493 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.309454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wr7\" (UniqueName: \"kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7\") pod \"maas-api-key-cleanup-29722800-959fl\" (UID: \"eceb335d-dead-4547-b3b0-4208e662d384\") " pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:00.455487 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.455423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:00.586089 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.586066 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722800-959fl"] Jul 06 20:00:00.588132 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:00:00.588105 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceb335d_dead_4547_b3b0_4208e662d384.slice/crio-30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb WatchSource:0}: Error finding container 30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb: Status 404 returned error can't find the container with id 30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb Jul 06 20:00:00.809398 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:00.809370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" event={"ID":"eceb335d-dead-4547-b3b0-4208e662d384","Type":"ContainerStarted","Data":"30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb"} Jul 06 20:00:01.812505 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:01.812474 2577 generic.go:356] "Generic (PLEG): container finished" podID="eceb335d-dead-4547-b3b0-4208e662d384" containerID="aa0541ed1683e4b972868088c01600dfc04df00bf74ea33ecc3e42b76704a87d" exitCode=0 Jul 06 20:00:01.812892 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:01.812565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" event={"ID":"eceb335d-dead-4547-b3b0-4208e662d384","Type":"ContainerDied","Data":"aa0541ed1683e4b972868088c01600dfc04df00bf74ea33ecc3e42b76704a87d"} Jul 06 20:00:02.310035 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:02.309997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:02.690464 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:02.690390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:02.789750 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:02.789723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:02.925525 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:02.925492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:02.971235 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:02.971208 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:03.022852 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.022823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6wr7\" (UniqueName: \"kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7\") pod \"eceb335d-dead-4547-b3b0-4208e662d384\" (UID: \"eceb335d-dead-4547-b3b0-4208e662d384\") " Jul 06 20:00:03.024828 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.024809 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7" (OuterVolumeSpecName: "kube-api-access-n6wr7") pod "eceb335d-dead-4547-b3b0-4208e662d384" (UID: "eceb335d-dead-4547-b3b0-4208e662d384"). InnerVolumeSpecName "kube-api-access-n6wr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:03.124417 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.124388 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wr7\" (UniqueName: \"kubernetes.io/projected/eceb335d-dead-4547-b3b0-4208e662d384-kube-api-access-n6wr7\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:00:03.239094 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.239068 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:03.823946 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.823915 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" Jul 06 20:00:03.823946 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.823932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722800-959fl" event={"ID":"eceb335d-dead-4547-b3b0-4208e662d384","Type":"ContainerDied","Data":"30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb"} Jul 06 20:00:03.824318 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:03.823979 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a90b0ae1afd8d9a65bc2e1e0e3cdf762dd3c44178e168022af6ce33b1d95cb" Jul 06 20:00:06.633324 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.633288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:06.705175 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.705142 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:00:06.705573 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.705558 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eceb335d-dead-4547-b3b0-4208e662d384" containerName="cleanup" Jul 06 20:00:06.705573 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.705574 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eceb335d-dead-4547-b3b0-4208e662d384" containerName="cleanup" Jul 06 20:00:06.705691 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.705645 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eceb335d-dead-4547-b3b0-4208e662d384" containerName="cleanup" Jul 06 20:00:06.709126 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.709107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.715158 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.715130 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 20:00:06.715470 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.715450 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-653e5c-serving-cert\"" Jul 06 20:00:06.722915 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.722879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:00:06.747316 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.747293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:06.754537 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.754514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5rn\" (UniqueName: \"kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.754655 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.754563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.754655 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.754640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.855472 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.855434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5rn\" (UniqueName: \"kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.855641 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.855478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.855919 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.855875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.856054 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:00:06.855887 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-653e5c-serving-cert: secret "maas-api-e2e-inf-a-653e5c-serving-cert" not found Jul 06 20:00:06.856112 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:00:06.856057 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls podName:9945870a-cad2-4749-838b-e7159d6bdc1d nodeName:}" failed. No retries permitted until 2026-07-06 20:00:07.356034966 +0000 UTC m=+1965.009602556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls") pod "maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" (UID: "9945870a-cad2-4749-838b-e7159d6bdc1d") : secret "maas-api-e2e-inf-a-653e5c-serving-cert" not found Jul 06 20:00:06.856483 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.856464 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:06.859878 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.859851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:06.873219 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:06.873198 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5rn\" (UniqueName: \"kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:07.075807 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.075776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:07.250381 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.250351 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:07.360919 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.360815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:07.363227 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.363208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:07.625867 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.625786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:07.773331 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.773303 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:00:07.775447 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:00:07.775416 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9945870a_cad2_4749_838b_e7159d6bdc1d.slice/crio-d1a83ef32f78e6fc4308b4a6610dcc87668110addc8e8a5aaf50f0df9458909d WatchSource:0}: Error finding container d1a83ef32f78e6fc4308b4a6610dcc87668110addc8e8a5aaf50f0df9458909d: Status 404 returned error can't find the container with id d1a83ef32f78e6fc4308b4a6610dcc87668110addc8e8a5aaf50f0df9458909d Jul 06 20:00:07.776825 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.776807 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:00:07.847614 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.847576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" event={"ID":"9945870a-cad2-4749-838b-e7159d6bdc1d","Type":"ContainerStarted","Data":"d1a83ef32f78e6fc4308b4a6610dcc87668110addc8e8a5aaf50f0df9458909d"} Jul 06 20:00:07.880585 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:07.880524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:08.855555 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:08.855524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" event={"ID":"9945870a-cad2-4749-838b-e7159d6bdc1d","Type":"ContainerStarted","Data":"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706"} Jul 06 20:00:08.855929 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:08.855641 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:08.886747 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:08.886699 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" podStartSLOduration=1.8801878909999998 podStartE2EDuration="2.886683525s" totalImagesPullingTime="1.006495634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:08.884715687 +0000 UTC m=+1966.538283285" watchObservedRunningTime="2026-07-06 20:00:08.886683525 +0000 UTC m=+1966.540251124" Jul 06 20:00:11.372603 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:11.372560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:11.715095 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:11.715015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:12.069874 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:12.069842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:12.405219 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:12.405129 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:14.866568 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:14.866494 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:00:15.160388 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.160300 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:15.242596 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.242562 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:00:15.246836 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.246812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.250076 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.250053 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-03ff21-serving-cert\"" Jul 06 20:00:15.273966 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.273942 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:00:15.284632 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.284596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:15.329812 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.329792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.329940 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.329827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxt8\" (UniqueName: \"kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.329940 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.329881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.407168 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.407144 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:15.431224 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.431161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.431224 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.431207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxt8\" (UniqueName: \"kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.431382 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.431284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.431382 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:00:15.431296 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-03ff21-serving-cert: secret "maas-api-e2e-inf-b-03ff21-serving-cert" not found Jul 06 20:00:15.431382 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:00:15.431354 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls podName:09d57bc9-03e2-42b8-8c72-b2b2089c1e9e nodeName:}" failed. No retries permitted until 2026-07-06 20:00:15.931340257 +0000 UTC m=+1973.584907834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls") pod "maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" (UID: "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e") : secret "maas-api-e2e-inf-b-03ff21-serving-cert" not found Jul 06 20:00:15.431945 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.431890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.455852 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.455824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxt8\" (UniqueName: \"kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.936104 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.936075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:15.938643 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:15.938617 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:16.158339 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:16.158308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:16.322339 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:16.322312 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:00:16.324392 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:00:16.324360 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d57bc9_03e2_42b8_8c72_b2b2089c1e9e.slice/crio-6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79 WatchSource:0}: Error finding container 6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79: Status 404 returned error can't find the container with id 6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79 Jul 06 20:00:16.395097 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:16.395070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:16.534560 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:16.534531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:16.898076 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:16.898051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" event={"ID":"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e","Type":"ContainerStarted","Data":"6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79"} Jul 06 20:00:17.905399 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:17.905363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" event={"ID":"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e","Type":"ContainerStarted","Data":"5fa205d0f458a9ca34091b8097b41955f8489c22b3121d23bbc13d124990e041"} Jul 06 20:00:17.905874 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:17.905425 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:17.930939 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:17.930868 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" podStartSLOduration=2.421765409 podStartE2EDuration="2.930853563s" totalImagesPullingTime="509.088154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:17.92955947 +0000 UTC m=+1975.583127069" watchObservedRunningTime="2026-07-06 20:00:17.930853563 +0000 UTC m=+1975.584421162" Jul 06 20:00:19.718851 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:19.718822 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:23.915559 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:23.915528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:00:28.956178 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:28.956139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:29.056184 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:29.056147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:30.025283 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:30.025252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:37.010347 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:37.010317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:37.111268 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:00:37.111238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:03.498226 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:03.498190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:23.013236 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:23.013204 2577 scope.go:117] "RemoveContainer" containerID="145bb7b1a2f06ff59825d58a209ae00ffea683cfb4bd4cea4a4a0c332b6f2924" Jul 06 20:01:24.348748 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:24.348708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:25.300636 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:25.300604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:25.490705 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:25.490675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:25.945733 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:25.945697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.046685 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.046655 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.146640 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.146606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.278852 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.278824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:01:26.279052 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.279029 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" podUID="9945870a-cad2-4749-838b-e7159d6bdc1d" containerName="maas-api" containerID="cri-o://676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706" gracePeriod=30 Jul 06 20:01:26.293244 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.293220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.532944 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.532884 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:01:26.637889 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.637860 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn5rn\" (UniqueName: \"kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn\") pod \"9945870a-cad2-4749-838b-e7159d6bdc1d\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " Jul 06 20:01:26.637889 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.637890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") pod \"9945870a-cad2-4749-838b-e7159d6bdc1d\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " Jul 06 20:01:26.638169 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.637953 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca\") pod \"9945870a-cad2-4749-838b-e7159d6bdc1d\" (UID: \"9945870a-cad2-4749-838b-e7159d6bdc1d\") " Jul 06 20:01:26.638370 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.638346 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9945870a-cad2-4749-838b-e7159d6bdc1d" (UID: "9945870a-cad2-4749-838b-e7159d6bdc1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:01:26.640079 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.640052 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn" (OuterVolumeSpecName: "kube-api-access-gn5rn") pod "9945870a-cad2-4749-838b-e7159d6bdc1d" (UID: "9945870a-cad2-4749-838b-e7159d6bdc1d"). InnerVolumeSpecName "kube-api-access-gn5rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:01:26.641233 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.641214 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9945870a-cad2-4749-838b-e7159d6bdc1d" (UID: "9945870a-cad2-4749-838b-e7159d6bdc1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:01:26.738697 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.738669 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5rn\" (UniqueName: \"kubernetes.io/projected/9945870a-cad2-4749-838b-e7159d6bdc1d-kube-api-access-gn5rn\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:01:26.738697 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.738695 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9945870a-cad2-4749-838b-e7159d6bdc1d-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:01:26.738836 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:26.738708 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9945870a-cad2-4749-838b-e7159d6bdc1d-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:01:27.292545 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.292511 2577 generic.go:356] "Generic (PLEG): container finished" podID="9945870a-cad2-4749-838b-e7159d6bdc1d" containerID="676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706" exitCode=0 Jul 06 20:01:27.292717 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.292552 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" Jul 06 20:01:27.292717 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.292556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" event={"ID":"9945870a-cad2-4749-838b-e7159d6bdc1d","Type":"ContainerDied","Data":"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706"} Jul 06 20:01:27.292717 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.292598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn" event={"ID":"9945870a-cad2-4749-838b-e7159d6bdc1d","Type":"ContainerDied","Data":"d1a83ef32f78e6fc4308b4a6610dcc87668110addc8e8a5aaf50f0df9458909d"} Jul 06 20:01:27.292717 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.292621 2577 scope.go:117] "RemoveContainer" containerID="676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706" Jul 06 20:01:27.301553 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.301532 2577 scope.go:117] "RemoveContainer" containerID="676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706" Jul 06 20:01:27.301803 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:01:27.301786 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706\": container with ID starting with 676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706 not found: ID does not exist" containerID="676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706" Jul 06 20:01:27.301855 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.301811 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706"} err="failed to get container status \"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706\": rpc error: code = NotFound desc = could not find container \"676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706\": container with ID starting with 676121510d60c47557a1daadad2e7cb60b89c8cf3f5801fe97b2f974a86aa706 not found: ID does not exist" Jul 06 20:01:27.310668 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.310648 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:01:27.314973 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:27.314944 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-653e5c-8546f84b44-jjmtn"] Jul 06 20:01:28.810128 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:28.810086 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9945870a-cad2-4749-838b-e7159d6bdc1d" path="/var/lib/kubelet/pods/9945870a-cad2-4749-838b-e7159d6bdc1d/volumes" Jul 06 20:01:31.014653 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:01:31.014622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:07.393869 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:07.393835 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:07.912155 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:07.912128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:08.215940 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:08.215895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:08.435161 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:08.435133 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:08.619060 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:08.618985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.052396 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.052358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.148235 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.148202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.286316 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.286283 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:02:09.286535 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.286505 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" podUID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" containerName="maas-api" containerID="cri-o://5fa205d0f458a9ca34091b8097b41955f8489c22b3121d23bbc13d124990e041" gracePeriod=30 Jul 06 20:02:09.316532 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.316510 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:09.527089 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.527056 2577 generic.go:356] "Generic (PLEG): container finished" podID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" containerID="5fa205d0f458a9ca34091b8097b41955f8489c22b3121d23bbc13d124990e041" exitCode=0 Jul 06 20:02:09.527401 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.527122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" event={"ID":"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e","Type":"ContainerDied","Data":"5fa205d0f458a9ca34091b8097b41955f8489c22b3121d23bbc13d124990e041"} Jul 06 20:02:09.527401 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.527198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" event={"ID":"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e","Type":"ContainerDied","Data":"6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79"} Jul 06 20:02:09.527401 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.527215 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fe4e370bc1923fd96d7613c7e22b4e21a880358c3870ee794d720d5aea8cc79" Jul 06 20:02:09.535064 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.535043 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:02:09.567253 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.567196 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqxt8\" (UniqueName: \"kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8\") pod \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " Jul 06 20:02:09.567360 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.567261 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") pod \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " Jul 06 20:02:09.567360 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.567309 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca\") pod \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\" (UID: \"09d57bc9-03e2-42b8-8c72-b2b2089c1e9e\") " Jul 06 20:02:09.567720 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.567692 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" (UID: "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:09.569123 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.569104 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8" (OuterVolumeSpecName: "kube-api-access-jqxt8") pod "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" (UID: "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e"). InnerVolumeSpecName "kube-api-access-jqxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:09.569290 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.569267 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" (UID: "09d57bc9-03e2-42b8-8c72-b2b2089c1e9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:02:09.668442 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.668409 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxt8\" (UniqueName: \"kubernetes.io/projected/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-kube-api-access-jqxt8\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:09.668442 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.668435 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:09.668442 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:09.668444 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:10.531954 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:10.531924 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72" Jul 06 20:02:10.554770 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:10.554739 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:02:10.559646 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:10.559618 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-03ff21-fddf48cdb-vph72"] Jul 06 20:02:10.809882 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:10.809812 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" path="/var/lib/kubelet/pods/09d57bc9-03e2-42b8-8c72-b2b2089c1e9e/volumes" Jul 06 20:02:14.939462 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:14.939434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:15.021108 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:15.021069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:15.232660 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:15.232625 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:15.486742 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:15.486665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:15.848220 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:15.848142 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.241388 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:16.241353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.680119 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:16.680040 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:18.790451 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.790419 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:18.864389 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864357 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:18.864751 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864736 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9945870a-cad2-4749-838b-e7159d6bdc1d" containerName="maas-api" Jul 06 20:02:18.864751 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864751 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9945870a-cad2-4749-838b-e7159d6bdc1d" containerName="maas-api" Jul 06 20:02:18.864911 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864767 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" containerName="maas-api" Jul 06 20:02:18.864911 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864780 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" containerName="maas-api" Jul 06 20:02:18.864911 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864838 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9945870a-cad2-4749-838b-e7159d6bdc1d" containerName="maas-api" Jul 06 20:02:18.864911 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.864848 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d57bc9-03e2-42b8-8c72-b2b2089c1e9e" containerName="maas-api" Jul 06 20:02:18.868245 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.868227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:18.871463 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.871441 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-f1eed6-serving-cert\"" Jul 06 20:02:18.871874 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.871852 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 20:02:18.878616 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.878570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:18.879584 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.879539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:18.951282 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.951256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:18.951446 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.951307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvsd\" (UniqueName: \"kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:18.951446 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:18.951340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.044400 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.044332 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.052424 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.052399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.052519 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.052464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.052519 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.052501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvsd\" (UniqueName: \"kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.052605 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:19.052554 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-f1eed6-serving-cert: secret "maas-api-e2e-api-a-f1eed6-serving-cert" not found Jul 06 20:02:19.052647 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:19.052633 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls podName:bb03abaf-06b0-49dd-b7b0-68b02183b27e nodeName:}" failed. No retries permitted until 2026-07-06 20:02:19.552613156 +0000 UTC m=+2097.206180741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls") pod "maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" (UID: "bb03abaf-06b0-49dd-b7b0-68b02183b27e") : secret "maas-api-e2e-api-a-f1eed6-serving-cert" not found Jul 06 20:02:19.053166 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.053148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.066332 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.066313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvsd\" (UniqueName: \"kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.180548 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.180520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.386147 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.386073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.485079 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.485053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.557323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.557292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.559590 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.559568 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") pod \"maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.780974 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.780943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:19.919541 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:19.919519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:19.921141 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:02:19.921116 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb03abaf_06b0_49dd_b7b0_68b02183b27e.slice/crio-beb7be22f4c95d5e5c6fcdfb735e1ce28e0ba01d2c406fb284697296a76c7873 WatchSource:0}: Error finding container beb7be22f4c95d5e5c6fcdfb735e1ce28e0ba01d2c406fb284697296a76c7873: Status 404 returned error can't find the container with id beb7be22f4c95d5e5c6fcdfb735e1ce28e0ba01d2c406fb284697296a76c7873 Jul 06 20:02:20.588343 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:20.588308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" event={"ID":"bb03abaf-06b0-49dd-b7b0-68b02183b27e","Type":"ContainerStarted","Data":"beb7be22f4c95d5e5c6fcdfb735e1ce28e0ba01d2c406fb284697296a76c7873"} Jul 06 20:02:21.594740 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:21.594696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" event={"ID":"bb03abaf-06b0-49dd-b7b0-68b02183b27e","Type":"ContainerStarted","Data":"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594"} Jul 06 20:02:21.595134 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:21.594802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:21.624374 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:21.624331 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" podStartSLOduration=2.900669742 podStartE2EDuration="3.624316943s" totalImagesPullingTime="723.647201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:21.621299668 +0000 UTC m=+2099.274867266" watchObservedRunningTime="2026-07-06 20:02:21.624316943 +0000 UTC m=+2099.277884542" Jul 06 20:02:23.363458 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:23.363364 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:23.881085 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:23.881058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:24.252660 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:24.252586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:24.759696 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:24.759664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:26.951816 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:26.951773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.052008 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.051978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.103839 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.103801 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:27.108249 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.108229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.111491 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.111471 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-0e21dc-serving-cert\"" Jul 06 20:02:27.118978 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.118957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:27.209757 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.209678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.227817 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.227790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.227950 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.227848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d9q\" (UniqueName: \"kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.227950 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.227912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.328581 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.328549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d9q\" (UniqueName: \"kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.328749 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.328589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.328749 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.328664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.328839 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:27.328787 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-0e21dc-serving-cert: secret "maas-api-e2e-api-b-0e21dc-serving-cert" not found Jul 06 20:02:27.328877 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:27.328861 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls podName:22ccec4e-539f-4806-ae36-dcbe6ee12256 nodeName:}" failed. No retries permitted until 2026-07-06 20:02:27.828839505 +0000 UTC m=+2105.482407088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls") pod "maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" (UID: "22ccec4e-539f-4806-ae36-dcbe6ee12256") : secret "maas-api-e2e-api-b-0e21dc-serving-cert" not found Jul 06 20:02:27.329321 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.329295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.341304 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.341274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d9q\" (UniqueName: \"kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.606427 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.606400 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:27.727623 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.727593 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.834240 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.834212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.836523 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.836504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") pod \"maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:27.881717 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.881653 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:27.976382 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:27.976355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:28.019453 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.019419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:28.158824 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.158801 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:28.160456 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:02:28.160431 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ccec4e_539f_4806_ae36_dcbe6ee12256.slice/crio-cc94aec778a25dadc1327e62a0837b9028114bd68aff766c413cf6550bdbf1f2 WatchSource:0}: Error finding container cc94aec778a25dadc1327e62a0837b9028114bd68aff766c413cf6550bdbf1f2: Status 404 returned error can't find the container with id cc94aec778a25dadc1327e62a0837b9028114bd68aff766c413cf6550bdbf1f2 Jul 06 20:02:28.635595 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.635562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" event={"ID":"22ccec4e-539f-4806-ae36-dcbe6ee12256","Type":"ContainerStarted","Data":"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94"} Jul 06 20:02:28.635694 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.635621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" event={"ID":"22ccec4e-539f-4806-ae36-dcbe6ee12256","Type":"ContainerStarted","Data":"cc94aec778a25dadc1327e62a0837b9028114bd68aff766c413cf6550bdbf1f2"} Jul 06 20:02:28.635781 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.635756 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:28.655404 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:28.655364 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" podStartSLOduration=1.309612119 podStartE2EDuration="1.655348446s" totalImagesPullingTime="345.736327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:28.654805376 +0000 UTC m=+2106.308372974" watchObservedRunningTime="2026-07-06 20:02:28.655348446 +0000 UTC m=+2106.308916046" Jul 06 20:02:31.413500 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:31.413436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:34.647654 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:34.647626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:38.037590 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:38.037555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:38.264543 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:38.264513 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:39.614950 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:39.614880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:40.032067 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.032018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:40.133814 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.133784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:40.357256 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.357174 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:40.357451 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.357368 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" podUID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" containerName="maas-api" containerID="cri-o://f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594" gracePeriod=30 Jul 06 20:02:40.381965 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.381940 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:40.611612 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.611553 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:40.705978 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.705930 2577 generic.go:356] "Generic (PLEG): container finished" podID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" containerID="f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594" exitCode=0 Jul 06 20:02:40.706364 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.706015 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" Jul 06 20:02:40.706364 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.706055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" event={"ID":"bb03abaf-06b0-49dd-b7b0-68b02183b27e","Type":"ContainerDied","Data":"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594"} Jul 06 20:02:40.706364 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.706105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8" event={"ID":"bb03abaf-06b0-49dd-b7b0-68b02183b27e","Type":"ContainerDied","Data":"beb7be22f4c95d5e5c6fcdfb735e1ce28e0ba01d2c406fb284697296a76c7873"} Jul 06 20:02:40.706364 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.706127 2577 scope.go:117] "RemoveContainer" containerID="f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594" Jul 06 20:02:40.715801 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.715782 2577 scope.go:117] "RemoveContainer" containerID="f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594" Jul 06 20:02:40.716056 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:40.716038 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594\": container with ID starting with f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594 not found: ID does not exist" containerID="f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594" Jul 06 20:02:40.716117 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.716066 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594"} err="failed to get container status \"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594\": rpc error: code = NotFound desc = could not find container \"f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594\": container with ID starting with f538bbf359170690f0256816f740dd91fbed8b5ce2411e8768b8084d44932594 not found: ID does not exist" Jul 06 20:02:40.746336 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.746312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca\") pod \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " Jul 06 20:02:40.746430 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.746361 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvsd\" (UniqueName: \"kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd\") pod \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " Jul 06 20:02:40.746500 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.746429 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") pod \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\" (UID: \"bb03abaf-06b0-49dd-b7b0-68b02183b27e\") " Jul 06 20:02:40.746659 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.746632 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb03abaf-06b0-49dd-b7b0-68b02183b27e" (UID: "bb03abaf-06b0-49dd-b7b0-68b02183b27e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:40.746730 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.746706 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb03abaf-06b0-49dd-b7b0-68b02183b27e-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:40.748372 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.748344 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd" (OuterVolumeSpecName: "kube-api-access-jwvsd") pod "bb03abaf-06b0-49dd-b7b0-68b02183b27e" (UID: "bb03abaf-06b0-49dd-b7b0-68b02183b27e"). InnerVolumeSpecName "kube-api-access-jwvsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:40.748474 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.748453 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb03abaf-06b0-49dd-b7b0-68b02183b27e" (UID: "bb03abaf-06b0-49dd-b7b0-68b02183b27e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:02:40.847304 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.847270 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvsd\" (UniqueName: \"kubernetes.io/projected/bb03abaf-06b0-49dd-b7b0-68b02183b27e-kube-api-access-jwvsd\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:40.847304 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:40.847301 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb03abaf-06b0-49dd-b7b0-68b02183b27e-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:41.026754 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:41.026725 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:41.029154 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:41.029128 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-f1eed6-66f74d7b7f-x8lh8"] Jul 06 20:02:42.809244 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:42.809214 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" path="/var/lib/kubelet/pods/bb03abaf-06b0-49dd-b7b0-68b02183b27e/volumes" Jul 06 20:02:46.963208 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:46.963176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:47.074475 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:47.074438 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:47.327139 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:47.327102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:47.528433 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:47.528404 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:47.705337 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:47.705237 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:48.110983 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.110947 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:48.216751 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.216717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:48.349065 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.349035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:48.480761 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.480732 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:48.480977 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.480955 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" podUID="22ccec4e-539f-4806-ae36-dcbe6ee12256" containerName="maas-api" containerID="cri-o://98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94" gracePeriod=30 Jul 06 20:02:48.533381 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.533357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:48.733098 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.733050 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:48.752091 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.752058 2577 generic.go:356] "Generic (PLEG): container finished" podID="22ccec4e-539f-4806-ae36-dcbe6ee12256" containerID="98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94" exitCode=0 Jul 06 20:02:48.752202 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.752095 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" Jul 06 20:02:48.752202 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.752133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" event={"ID":"22ccec4e-539f-4806-ae36-dcbe6ee12256","Type":"ContainerDied","Data":"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94"} Jul 06 20:02:48.752202 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.752184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs" event={"ID":"22ccec4e-539f-4806-ae36-dcbe6ee12256","Type":"ContainerDied","Data":"cc94aec778a25dadc1327e62a0837b9028114bd68aff766c413cf6550bdbf1f2"} Jul 06 20:02:48.752310 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.752208 2577 scope.go:117] "RemoveContainer" containerID="98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94" Jul 06 20:02:48.760427 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.760409 2577 scope.go:117] "RemoveContainer" containerID="98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94" Jul 06 20:02:48.760679 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:02:48.760662 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94\": container with ID starting with 98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94 not found: ID does not exist" containerID="98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94" Jul 06 20:02:48.760725 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.760686 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94"} err="failed to get container status \"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94\": rpc error: code = NotFound desc = could not find container \"98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94\": container with ID starting with 98df5fb9580d2a11dc934df804bef09034cad2475ac0c59f027f00914dff1e94 not found: ID does not exist" Jul 06 20:02:48.805469 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.805449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca\") pod \"22ccec4e-539f-4806-ae36-dcbe6ee12256\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " Jul 06 20:02:48.805594 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.805484 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87d9q\" (UniqueName: \"kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q\") pod \"22ccec4e-539f-4806-ae36-dcbe6ee12256\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " Jul 06 20:02:48.805594 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.805525 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") pod \"22ccec4e-539f-4806-ae36-dcbe6ee12256\" (UID: \"22ccec4e-539f-4806-ae36-dcbe6ee12256\") " Jul 06 20:02:48.805785 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.805762 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22ccec4e-539f-4806-ae36-dcbe6ee12256" (UID: "22ccec4e-539f-4806-ae36-dcbe6ee12256"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:02:48.807530 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.807509 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q" (OuterVolumeSpecName: "kube-api-access-87d9q") pod "22ccec4e-539f-4806-ae36-dcbe6ee12256" (UID: "22ccec4e-539f-4806-ae36-dcbe6ee12256"). InnerVolumeSpecName "kube-api-access-87d9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:02:48.807737 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.807718 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22ccec4e-539f-4806-ae36-dcbe6ee12256" (UID: "22ccec4e-539f-4806-ae36-dcbe6ee12256"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:02:48.906330 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.906300 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22ccec4e-539f-4806-ae36-dcbe6ee12256-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:48.906330 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.906328 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccec4e-539f-4806-ae36-dcbe6ee12256-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:48.906330 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:48.906338 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87d9q\" (UniqueName: \"kubernetes.io/projected/22ccec4e-539f-4806-ae36-dcbe6ee12256-kube-api-access-87d9q\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:02:49.072459 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:49.072389 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:49.076940 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:49.076886 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-0e21dc-6697768f48-lwrrs"] Jul 06 20:02:50.810188 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:50.810152 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ccec4e-539f-4806-ae36-dcbe6ee12256" path="/var/lib/kubelet/pods/22ccec4e-539f-4806-ae36-dcbe6ee12256/volumes" Jul 06 20:02:54.575594 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:54.575560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:54.653680 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:54.653643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:54.902572 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:54.902494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:55.099640 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:55.099609 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:55.475491 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:55.475456 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:55.829329 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:55.829246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:56.509374 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:56.509338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.037030 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.036983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.109870 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.109835 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:02:59.110405 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110388 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" containerName="maas-api" Jul 06 20:02:59.110493 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110407 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" containerName="maas-api" Jul 06 20:02:59.110493 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110434 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ccec4e-539f-4806-ae36-dcbe6ee12256" containerName="maas-api" Jul 06 20:02:59.110493 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110442 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ccec4e-539f-4806-ae36-dcbe6ee12256" containerName="maas-api" Jul 06 20:02:59.110671 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110546 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ccec4e-539f-4806-ae36-dcbe6ee12256" containerName="maas-api" Jul 06 20:02:59.110671 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.110560 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb03abaf-06b0-49dd-b7b0-68b02183b27e" containerName="maas-api" Jul 06 20:02:59.114051 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.114031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.116864 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.116846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-4430fb-serving-cert\"" Jul 06 20:02:59.117060 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.117042 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-vsm6f\"" Jul 06 20:02:59.124856 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.124831 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:02:59.127519 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.127492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.172802 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.172764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqkj\" (UniqueName: \"kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.173005 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.172853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.173005 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.172887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.207021 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.206991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.273350 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.273316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.273350 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.273354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.273615 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.273403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqkj\" (UniqueName: \"kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.274004 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.273980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.277016 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.276986 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.290982 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.290892 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqkj\" (UniqueName: \"kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj\") pod \"maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.304144 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.304121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.426286 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.426238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:02:59.569740 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.569715 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:02:59.571257 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:02:59.571227 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f66d47_305a_4910_9db6_4636db606a80.slice/crio-ddee55c49c4157cb7dbd12a6ac99350022d0113eb0b47e09d8eea512e82dffb5 WatchSource:0}: Error finding container ddee55c49c4157cb7dbd12a6ac99350022d0113eb0b47e09d8eea512e82dffb5: Status 404 returned error can't find the container with id ddee55c49c4157cb7dbd12a6ac99350022d0113eb0b47e09d8eea512e82dffb5 Jul 06 20:02:59.689521 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.689491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.801201 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.801169 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:59.813254 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:02:59.813225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" event={"ID":"64f66d47-305a-4910-9db6-4636db606a80","Type":"ContainerStarted","Data":"ddee55c49c4157cb7dbd12a6ac99350022d0113eb0b47e09d8eea512e82dffb5"} Jul 06 20:03:00.819235 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:00.819202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" event={"ID":"64f66d47-305a-4910-9db6-4636db606a80","Type":"ContainerStarted","Data":"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7"} Jul 06 20:03:00.819623 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:00.819265 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:03:03.573646 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:03.573581 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:03.944258 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:03.944165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:04.296323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:04.296260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:04.704551 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:04.704471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:06.829301 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:06.829276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:03:06.851110 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:06.851058 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" podStartSLOduration=7.404720755 podStartE2EDuration="7.851043378s" totalImagesPullingTime="446.322623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:00.853449073 +0000 UTC m=+2138.507016673" watchObservedRunningTime="2026-07-06 20:03:06.851043378 +0000 UTC m=+2144.504610976" Jul 06 20:03:07.055807 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.055768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.116619 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.116529 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:03:07.120844 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.120819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.124453 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.124425 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-33ae61-serving-cert\"" Jul 06 20:03:07.136730 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.136702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:03:07.138595 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.138570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.138794 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.138767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.138966 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.138885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.155042 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.155020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.228911 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.228858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.240169 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.240142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.240425 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:03:07.240404 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-33ae61-serving-cert: secret "maas-api-e2e-shared-b-33ae61-serving-cert" not found Jul 06 20:03:07.240558 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.240525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.240662 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:03:07.240649 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls podName:39844943-34d6-49c7-ac10-8c1440109494 nodeName:}" failed. No retries permitted until 2026-07-06 20:03:07.740548316 +0000 UTC m=+2145.394115910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls") pod "maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" (UID: "39844943-34d6-49c7-ac10-8c1440109494") : secret "maas-api-e2e-shared-b-33ae61-serving-cert" not found Jul 06 20:03:07.240851 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.240826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.241338 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.241320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.264667 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.264640 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.499852 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.499793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.654844 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.654799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.744775 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.744738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:07.746635 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.746607 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.747381 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:07.747361 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") pod \"maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:08.032924 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.032851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:08.185839 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.185807 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:03:08.186868 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:03:08.186830 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39844943_34d6_49c7_ac10_8c1440109494.slice/crio-399adf582e318e942f56852a6a5b9d4f3714ead94a6d76f6d879305062616dd0 WatchSource:0}: Error finding container 399adf582e318e942f56852a6a5b9d4f3714ead94a6d76f6d879305062616dd0: Status 404 returned error can't find the container with id 399adf582e318e942f56852a6a5b9d4f3714ead94a6d76f6d879305062616dd0 Jul 06 20:03:08.867881 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.867788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" event={"ID":"39844943-34d6-49c7-ac10-8c1440109494","Type":"ContainerStarted","Data":"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277"} Jul 06 20:03:08.867881 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.867842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" event={"ID":"39844943-34d6-49c7-ac10-8c1440109494","Type":"ContainerStarted","Data":"399adf582e318e942f56852a6a5b9d4f3714ead94a6d76f6d879305062616dd0"} Jul 06 20:03:08.868131 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.867925 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:08.893887 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:08.893839 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" podStartSLOduration=1.566270588 podStartE2EDuration="1.893825458s" totalImagesPullingTime="327.55487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:08.890945443 +0000 UTC m=+2146.544513044" watchObservedRunningTime="2026-07-06 20:03:08.893825458 +0000 UTC m=+2146.547393057" Jul 06 20:03:11.631100 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:11.631066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:14.879405 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:14.879330 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:03:18.139065 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:18.139013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:20.545256 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:20.545219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:20.647856 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:20.647815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:28.605488 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:28.605447 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:28.738867 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:28.738833 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:41.932703 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:41.932661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:52.205304 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:03:52.205265 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:49.083423 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:05:49.083392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:12.384462 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:12.384423 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:13.309775 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:13.309734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:20.697570 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:20.697481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:20.756388 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:20.756351 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:23.085326 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:23.085296 2577 scope.go:117] "RemoveContainer" containerID="5fa205d0f458a9ca34091b8097b41955f8489c22b3121d23bbc13d124990e041" Jul 06 20:06:23.789309 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:23.789277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:28.868547 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:28.868507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.482576 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:06:49.482538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:24.660927 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:24.660880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.626840 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:25.626806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:27.492527 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:27.492494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:39.092592 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:39.092553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:39.184225 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:39.184192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:51.559025 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:07:51.558929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:21.309093 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:21.309060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:21.408381 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:21.408342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:29.419350 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:29.419308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:29.521130 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:29.521091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:30.644605 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:30.644566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:31.502104 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:31.502066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:32.799853 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:32.799816 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:33.247412 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.247378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:33.347738 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.347695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:33.449607 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.449574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:33.543974 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.543879 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:08:33.544170 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.544134 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" podUID="64f66d47-305a-4910-9db6-4636db606a80" containerName="maas-api" containerID="cri-o://918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7" gracePeriod=30 Jul 06 20:08:33.576043 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.576019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:33.803719 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.803658 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:08:33.950094 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.950066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls\") pod \"64f66d47-305a-4910-9db6-4636db606a80\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " Jul 06 20:08:33.950258 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.950159 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zqkj\" (UniqueName: \"kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj\") pod \"64f66d47-305a-4910-9db6-4636db606a80\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " Jul 06 20:08:33.950258 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.950214 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca\") pod \"64f66d47-305a-4910-9db6-4636db606a80\" (UID: \"64f66d47-305a-4910-9db6-4636db606a80\") " Jul 06 20:08:33.950611 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.950581 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64f66d47-305a-4910-9db6-4636db606a80" (UID: "64f66d47-305a-4910-9db6-4636db606a80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:08:33.952423 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.952395 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj" (OuterVolumeSpecName: "kube-api-access-9zqkj") pod "64f66d47-305a-4910-9db6-4636db606a80" (UID: "64f66d47-305a-4910-9db6-4636db606a80"). InnerVolumeSpecName "kube-api-access-9zqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:33.952521 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:33.952450 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64f66d47-305a-4910-9db6-4636db606a80" (UID: "64f66d47-305a-4910-9db6-4636db606a80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:08:34.051628 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.051606 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zqkj\" (UniqueName: \"kubernetes.io/projected/64f66d47-305a-4910-9db6-4636db606a80-kube-api-access-9zqkj\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:08:34.051628 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.051629 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f66d47-305a-4910-9db6-4636db606a80-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:08:34.051777 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.051640 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64f66d47-305a-4910-9db6-4636db606a80-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:08:34.637155 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.637118 2577 generic.go:356] "Generic (PLEG): container finished" podID="64f66d47-305a-4910-9db6-4636db606a80" containerID="918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7" exitCode=0 Jul 06 20:08:34.637332 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.637169 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" Jul 06 20:08:34.637332 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.637201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" event={"ID":"64f66d47-305a-4910-9db6-4636db606a80","Type":"ContainerDied","Data":"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7"} Jul 06 20:08:34.637332 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.637255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g" event={"ID":"64f66d47-305a-4910-9db6-4636db606a80","Type":"ContainerDied","Data":"ddee55c49c4157cb7dbd12a6ac99350022d0113eb0b47e09d8eea512e82dffb5"} Jul 06 20:08:34.637332 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.637279 2577 scope.go:117] "RemoveContainer" containerID="918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7" Jul 06 20:08:34.646797 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.646782 2577 scope.go:117] "RemoveContainer" containerID="918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7" Jul 06 20:08:34.647063 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:08:34.647045 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7\": container with ID starting with 918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7 not found: ID does not exist" containerID="918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7" Jul 06 20:08:34.647132 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.647073 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7"} err="failed to get container status \"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7\": rpc error: code = NotFound desc = could not find container \"918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7\": container with ID starting with 918404bc8df6336a712852c6324dc02ba1454ca73f195497efa204e851fe09b7 not found: ID does not exist" Jul 06 20:08:34.670778 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.670753 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:08:34.674604 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.674580 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-4430fb-77fcf69c9d-lx49g"] Jul 06 20:08:34.810308 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:34.810277 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f66d47-305a-4910-9db6-4636db606a80" path="/var/lib/kubelet/pods/64f66d47-305a-4910-9db6-4636db606a80/volumes" Jul 06 20:08:38.326470 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:38.326436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:51.497862 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:51.497826 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:51.572894 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:08:51.572867 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:15.507283 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:15.507198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:15.786322 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:15.786235 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:16.031569 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.031527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:16.261806 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.261769 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:16.649587 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.649494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:16.750046 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.750010 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:16.895190 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.895150 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:09:16.895426 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.895401 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" podUID="39844943-34d6-49c7-ac10-8c1440109494" containerName="maas-api" containerID="cri-o://48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277" gracePeriod=30 Jul 06 20:09:16.919308 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:16.919280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:17.137131 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.137111 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:09:17.191835 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.191769 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt\") pod \"39844943-34d6-49c7-ac10-8c1440109494\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " Jul 06 20:09:17.191977 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.191850 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca\") pod \"39844943-34d6-49c7-ac10-8c1440109494\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " Jul 06 20:09:17.191977 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.191892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") pod \"39844943-34d6-49c7-ac10-8c1440109494\" (UID: \"39844943-34d6-49c7-ac10-8c1440109494\") " Jul 06 20:09:17.192208 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.192186 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39844943-34d6-49c7-ac10-8c1440109494" (UID: "39844943-34d6-49c7-ac10-8c1440109494"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:09:17.193888 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.193867 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39844943-34d6-49c7-ac10-8c1440109494" (UID: "39844943-34d6-49c7-ac10-8c1440109494"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:09:17.193965 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.193947 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt" (OuterVolumeSpecName: "kube-api-access-xnhvt") pod "39844943-34d6-49c7-ac10-8c1440109494" (UID: "39844943-34d6-49c7-ac10-8c1440109494"). InnerVolumeSpecName "kube-api-access-xnhvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:09:17.292506 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.292477 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39844943-34d6-49c7-ac10-8c1440109494-maas-api-tls\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:09:17.292506 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.292500 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnhvt\" (UniqueName: \"kubernetes.io/projected/39844943-34d6-49c7-ac10-8c1440109494-kube-api-access-xnhvt\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:09:17.292506 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.292510 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39844943-34d6-49c7-ac10-8c1440109494-openshift-service-ca\") on node \"ip-10-0-130-143.ec2.internal\" DevicePath \"\"" Jul 06 20:09:17.877890 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.877858 2577 generic.go:356] "Generic (PLEG): container finished" podID="39844943-34d6-49c7-ac10-8c1440109494" containerID="48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277" exitCode=0 Jul 06 20:09:17.878305 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.877916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" event={"ID":"39844943-34d6-49c7-ac10-8c1440109494","Type":"ContainerDied","Data":"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277"} Jul 06 20:09:17.878305 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.877920 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" Jul 06 20:09:17.878305 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.877956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5" event={"ID":"39844943-34d6-49c7-ac10-8c1440109494","Type":"ContainerDied","Data":"399adf582e318e942f56852a6a5b9d4f3714ead94a6d76f6d879305062616dd0"} Jul 06 20:09:17.878305 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.877977 2577 scope.go:117] "RemoveContainer" containerID="48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277" Jul 06 20:09:17.887681 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.887664 2577 scope.go:117] "RemoveContainer" containerID="48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277" Jul 06 20:09:17.887958 ip-10-0-130-143 kubenswrapper[2577]: E0706 20:09:17.887940 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277\": container with ID starting with 48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277 not found: ID does not exist" containerID="48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277" Jul 06 20:09:17.888035 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.887969 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277"} err="failed to get container status \"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277\": rpc error: code = NotFound desc = could not find container \"48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277\": container with ID starting with 48032cf8618ed7373860ca7c2787314a3d632ec3b730415374921ca8da688277 not found: ID does not exist" Jul 06 20:09:17.900309 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.900286 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:09:17.907062 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:17.907039 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-33ae61-8769cdb85-f7js5"] Jul 06 20:09:18.809465 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:18.809434 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39844943-34d6-49c7-ac10-8c1440109494" path="/var/lib/kubelet/pods/39844943-34d6-49c7-ac10-8c1440109494/volumes" Jul 06 20:09:22.707668 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:22.707587 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:22.807449 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:22.807409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:22.936125 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:22.936090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:23.032638 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:23.032606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:23.216238 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:23.216209 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:28.597308 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:28.597262 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tw2sp_1d278012-f523-4671-8938-ee734022253d/manager/0.log" Jul 06 20:09:28.727494 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:28.727442 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zqvrr_cec36c91-f612-408f-8513-18472a8348b7/manager/0.log" Jul 06 20:09:28.854084 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:28.853999 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7c874b949-m545l_1a5c315d-e4df-4aff-87d4-cd288dc5aa95/maas-api/0.log" Jul 06 20:09:28.966794 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:28.966750 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-dpl42_77083555-2da8-439a-b958-a1233b7c0975/cleanup/0.log" Jul 06 20:09:29.086970 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:29.086887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-959fl_eceb335d-dead-4547-b3b0-4208e662d384/cleanup/0.log" Jul 06 20:09:29.219116 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:29.219088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b8f5864f-n29mx_7e12732d-bc55-4abb-a28b-f2b5bdbc4188/manager/0.log" Jul 06 20:09:29.702045 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:29.701968 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b585888c-b9qqp_0604ee50-a8e0-44d2-bc77-eae76ed81f19/copy-manifests/0.log" Jul 06 20:09:29.719798 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:29.719771 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b585888c-b9qqp_0604ee50-a8e0-44d2-bc77-eae76ed81f19/manager/0.log" Jul 06 20:09:29.969349 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:29.969312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-fh2jn_4143ea7c-27eb-490c-83a1-fd6b025e6c38/postgres/0.log" Jul 06 20:09:30.814871 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.814843 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/util/0.log" Jul 06 20:09:30.822072 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.822049 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/pull/0.log" Jul 06 20:09:30.829163 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.829142 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/extract/0.log" Jul 06 20:09:30.947778 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.947750 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/util/0.log" Jul 06 20:09:30.955183 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.955159 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/pull/0.log" Jul 06 20:09:30.964231 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:30.964198 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/extract/0.log" Jul 06 20:09:31.084203 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.084133 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/pull/0.log" Jul 06 20:09:31.090952 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.090919 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/extract/0.log" Jul 06 20:09:31.097439 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.097415 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/util/0.log" Jul 06 20:09:31.220609 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.220591 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/util/0.log" Jul 06 20:09:31.228233 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.228212 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/pull/0.log" Jul 06 20:09:31.235730 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.235712 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/extract/0.log" Jul 06 20:09:31.502782 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.502746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qqngq_468797fc-f8ed-4b0b-9110-8db59469ddc6/manager/0.log" Jul 06 20:09:31.630821 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.630795 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b2m9w_25cd4103-60d0-4cb7-9295-5a664accc94f/manager/0.log" Jul 06 20:09:31.761408 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.761339 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7wx8q_9cca4385-a358-42b3-aac7-0a538a7724ae/kuadrant-console-plugin/0.log" Jul 06 20:09:31.898645 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:31.898609 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xjpbx_25a32fa0-eb79-4a67-9574-d89640ddca9c/registry-server/0.log" Jul 06 20:09:32.038735 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:32.038665 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fw6w2_7fea671f-f8a2-4240-a917-9332ce208f4a/manager/0.log" Jul 06 20:09:32.297638 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:32.297556 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c5dv2_4ef0ccf7-0681-493e-9306-54985188afd6/manager/0.log" Jul 06 20:09:32.936563 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:32.936535 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f6467984-49ldb_19a5ba3d-428d-4fd0-82e3-f45a8f81901c/kube-auth-proxy/0.log" Jul 06 20:09:33.449715 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:33.449663 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-hr9tb_443768c2-5343-482b-ad62-77576ef6d2aa/payload-processing/0.log" Jul 06 20:09:33.929778 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:33.929746 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ldr6w_f983db50-3c59-43a9-a64f-18912819a236/main/0.log" Jul 06 20:09:34.292846 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:34.292820 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtqd9q_88f7b67e-dd33-4386-8deb-1a93456194b5/main/0.log" Jul 06 20:09:38.972449 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:38.972400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:39.083769 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:39.083731 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:41.722348 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:41.722317 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-65kqw_d11926aa-a0cc-40d1-9c68-17a44704d1b6/global-pull-secret-syncer/0.log" Jul 06 20:09:41.863806 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:41.863776 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7qgxs_ddf894a6-f277-4b6b-8306-ab44bdce81bf/konnectivity-agent/0.log" Jul 06 20:09:41.973115 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:41.973006 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-143.ec2.internal_aa627ad35b77ae9bde45172877976b6e/haproxy/0.log" Jul 06 20:09:46.426128 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.426056 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/extract/0.log" Jul 06 20:09:46.453879 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.453851 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/util/0.log" Jul 06 20:09:46.482182 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.482153 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bbfn7_8a3d529c-a24b-4c0e-9dc0-f4e75df1011c/pull/0.log" Jul 06 20:09:46.510257 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.510235 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/extract/0.log" Jul 06 20:09:46.534195 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.534178 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/util/0.log" Jul 06 20:09:46.560863 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.560844 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rljtt_20f5a4e4-6f1b-4497-8af3-cd8dfab1a8bd/pull/0.log" Jul 06 20:09:46.590338 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.590319 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/extract/0.log" Jul 06 20:09:46.617688 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.617663 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/util/0.log" Jul 06 20:09:46.645025 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.645003 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nmkzw_a67c54c6-ecc3-4458-b669-af3ed1bd3d50/pull/0.log" Jul 06 20:09:46.675547 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.675521 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/extract/0.log" Jul 06 20:09:46.696647 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.696593 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/util/0.log" Jul 06 20:09:46.730141 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:46.730121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1z8n7x_5e29a6c8-cb58-4309-99f0-e27993b54516/pull/0.log" Jul 06 20:09:47.144529 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.144500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qqngq_468797fc-f8ed-4b0b-9110-8db59469ddc6/manager/0.log" Jul 06 20:09:47.168834 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.168809 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b2m9w_25cd4103-60d0-4cb7-9295-5a664accc94f/manager/0.log" Jul 06 20:09:47.196238 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.196215 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7wx8q_9cca4385-a358-42b3-aac7-0a538a7724ae/kuadrant-console-plugin/0.log" Jul 06 20:09:47.247062 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.247039 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xjpbx_25a32fa0-eb79-4a67-9574-d89640ddca9c/registry-server/0.log" Jul 06 20:09:47.453836 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.453750 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fw6w2_7fea671f-f8a2-4240-a917-9332ce208f4a/manager/0.log" Jul 06 20:09:47.579125 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:47.579094 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c5dv2_4ef0ccf7-0681-493e-9306-54985188afd6/manager/0.log" Jul 06 20:09:49.810070 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:49.810040 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4p9t_d88ddcf6-b088-43d1-aff7-249563ad1093/node-exporter/0.log" Jul 06 20:09:49.831984 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:49.831958 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4p9t_d88ddcf6-b088-43d1-aff7-249563ad1093/kube-rbac-proxy/0.log" Jul 06 20:09:49.859756 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:49.859734 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4p9t_d88ddcf6-b088-43d1-aff7-249563ad1093/init-textfile/0.log" Jul 06 20:09:50.295370 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:50.295340 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-zxgkj_a72010f0-fecf-4ae9-b1ae-a63dd896f581/prometheus-operator-admission-webhook/0.log" Jul 06 20:09:51.730096 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.730064 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-l7gvl_de34a387-7a02-4be4-8c42-aba4d0989fb1/networking-console-plugin/0.log" Jul 06 20:09:51.768914 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.768878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf"] Jul 06 20:09:51.769250 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769237 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39844943-34d6-49c7-ac10-8c1440109494" containerName="maas-api" Jul 06 20:09:51.769301 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769251 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="39844943-34d6-49c7-ac10-8c1440109494" containerName="maas-api" Jul 06 20:09:51.769301 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769265 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f66d47-305a-4910-9db6-4636db606a80" containerName="maas-api" Jul 06 20:09:51.769301 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769272 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f66d47-305a-4910-9db6-4636db606a80" containerName="maas-api" Jul 06 20:09:51.769412 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f66d47-305a-4910-9db6-4636db606a80" containerName="maas-api" Jul 06 20:09:51.769412 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.769349 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="39844943-34d6-49c7-ac10-8c1440109494" containerName="maas-api" Jul 06 20:09:51.772460 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.772444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.775554 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.775434 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f6s5f\"/\"kube-root-ca.crt\"" Jul 06 20:09:51.775697 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.775593 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f6s5f\"/\"default-dockercfg-p6z85\"" Jul 06 20:09:51.776718 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.776703 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f6s5f\"/\"openshift-service-ca.crt\"" Jul 06 20:09:51.781088 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.781068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf"] Jul 06 20:09:51.857259 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.857236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-proc\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.857386 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.857267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567fn\" (UniqueName: \"kubernetes.io/projected/79aa706c-dd41-43f9-933e-f900f5624221-kube-api-access-567fn\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.857386 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.857290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-podres\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.857462 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.857399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-lib-modules\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.857462 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.857430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-sys\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.957985 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.957959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-lib-modules\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958116 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.957997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-sys\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958116 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-proc\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958116 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-sys\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958116 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567fn\" (UniqueName: \"kubernetes.io/projected/79aa706c-dd41-43f9-933e-f900f5624221-kube-api-access-567fn\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-lib-modules\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-podres\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958177 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-proc\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.958323 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.958302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79aa706c-dd41-43f9-933e-f900f5624221-podres\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:51.967126 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:51.967105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-567fn\" (UniqueName: \"kubernetes.io/projected/79aa706c-dd41-43f9-933e-f900f5624221-kube-api-access-567fn\") pod \"perf-node-gather-daemonset-p44kf\" (UID: \"79aa706c-dd41-43f9-933e-f900f5624221\") " pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:52.082937 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:52.082868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:52.212761 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:52.212730 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf"] Jul 06 20:09:52.213981 ip-10-0-130-143 kubenswrapper[2577]: W0706 20:09:52.213951 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79aa706c_dd41_43f9_933e_f900f5624221.slice/crio-537b975b987b4e8394e445dbeb4debc52bd6d96ed8e737d9c8e9abe4dadf8cff WatchSource:0}: Error finding container 537b975b987b4e8394e445dbeb4debc52bd6d96ed8e737d9c8e9abe4dadf8cff: Status 404 returned error can't find the container with id 537b975b987b4e8394e445dbeb4debc52bd6d96ed8e737d9c8e9abe4dadf8cff Jul 06 20:09:52.969380 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:52.969336 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9228p_0061534e-5e79-49dc-917e-1036704ca034/download-server/0.log" Jul 06 20:09:53.066791 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:53.066756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" event={"ID":"79aa706c-dd41-43f9-933e-f900f5624221","Type":"ContainerStarted","Data":"004178ed785ec543a74678c398d474d39afad28a0980d91d86caf1161f77c48a"} Jul 06 20:09:53.066989 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:53.066806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" event={"ID":"79aa706c-dd41-43f9-933e-f900f5624221","Type":"ContainerStarted","Data":"537b975b987b4e8394e445dbeb4debc52bd6d96ed8e737d9c8e9abe4dadf8cff"} Jul 06 20:09:53.066989 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:53.066835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:09:53.084425 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:53.084384 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" podStartSLOduration=2.084369919 podStartE2EDuration="2.084369919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:53.083517065 +0000 UTC m=+2550.737084665" watchObservedRunningTime="2026-07-06 20:09:53.084369919 +0000 UTC m=+2550.737937517" Jul 06 20:09:54.722737 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:54.722706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nxzj_fadcafe9-3b20-40c4-b1a8-15860c940eec/dns/0.log" Jul 06 20:09:54.748534 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:54.748510 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nxzj_fadcafe9-3b20-40c4-b1a8-15860c940eec/kube-rbac-proxy/0.log" Jul 06 20:09:54.952125 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:54.952101 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xkrj6_a61d4a93-c673-46d3-a5c1-bc5dd06c8c7a/dns-node-resolver/0.log" Jul 06 20:09:55.540473 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:55.540435 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cf4c59977-qr592_9b20d746-f135-428f-bee1-f662be0b9174/registry/0.log" Jul 06 20:09:55.593527 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:55.593473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7z4x5_7ca5c635-0f15-4354-a416-513cbc89b7bb/node-ca/0.log" Jul 06 20:09:56.692872 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:56.692841 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f6467984-49ldb_19a5ba3d-428d-4fd0-82e3-f45a8f81901c/kube-auth-proxy/0.log" Jul 06 20:09:56.888472 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:56.888442 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-hr9tb_443768c2-5343-482b-ad62-77576ef6d2aa/payload-processing/0.log" Jul 06 20:09:57.622621 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:57.622589 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5dgxw_8f6910e1-7652-4261-a307-3fbdc052ff9e/serve-healthcheck-canary/0.log" Jul 06 20:09:58.259892 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:58.259864 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clz5w_db58b7de-f080-402f-9ad4-e50c230c3e8f/kube-rbac-proxy/0.log" Jul 06 20:09:58.282830 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:58.282803 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clz5w_db58b7de-f080-402f-9ad4-e50c230c3e8f/exporter/0.log" Jul 06 20:09:58.307374 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:58.307351 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-clz5w_db58b7de-f080-402f-9ad4-e50c230c3e8f/extractor/0.log" Jul 06 20:09:59.081827 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:09:59.081803 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f6s5f/perf-node-gather-daemonset-p44kf" Jul 06 20:10:00.647873 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:00.647842 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tw2sp_1d278012-f523-4671-8938-ee734022253d/manager/0.log" Jul 06 20:10:00.887250 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:00.887218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zqvrr_cec36c91-f612-408f-8513-18472a8348b7/manager/0.log" Jul 06 20:10:00.939558 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:00.939537 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7c874b949-m545l_1a5c315d-e4df-4aff-87d4-cd288dc5aa95/maas-api/0.log" Jul 06 20:10:00.960744 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:00.960716 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-dpl42_77083555-2da8-439a-b958-a1233b7c0975/cleanup/0.log" Jul 06 20:10:00.988518 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:00.988441 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722800-959fl_eceb335d-dead-4547-b3b0-4208e662d384/cleanup/0.log" Jul 06 20:10:01.153537 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:01.153510 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b8f5864f-n29mx_7e12732d-bc55-4abb-a28b-f2b5bdbc4188/manager/0.log" Jul 06 20:10:01.480252 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:01.480223 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b585888c-b9qqp_0604ee50-a8e0-44d2-bc77-eae76ed81f19/manager/0.log" Jul 06 20:10:01.499890 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:01.499864 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57b585888c-b9qqp_0604ee50-a8e0-44d2-bc77-eae76ed81f19/copy-manifests/0.log" Jul 06 20:10:01.608539 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:01.608481 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-fh2jn_4143ea7c-27eb-490c-83a1-fd6b025e6c38/postgres/0.log" Jul 06 20:10:02.602947 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:02.602917 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f6467984-49ldb_19a5ba3d-428d-4fd0-82e3-f45a8f81901c/kube-auth-proxy/0.log" Jul 06 20:10:02.760330 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:02.760292 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-hr9tb_443768c2-5343-482b-ad62-77576ef6d2aa/payload-processing/0.log" Jul 06 20:10:04.142646 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:04.142617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58f6467984-49ldb_19a5ba3d-428d-4fd0-82e3-f45a8f81901c/kube-auth-proxy/0.log" Jul 06 20:10:04.301844 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:04.301817 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-hr9tb_443768c2-5343-482b-ad62-77576ef6d2aa/payload-processing/0.log" Jul 06 20:10:08.052429 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:08.052399 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-lk95d_c7f75bce-0a16-4a12-b137-cbb82dd0b5d7/migrator/0.log" Jul 06 20:10:08.074477 ip-10-0-130-143 kubenswrapper[2577]: I0706 20:10:08.074453 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-lk95d_c7f75bce-0a16-4a12-b137-cbb82dd0b5d7/graceful-termination/0.log"