Jul 07 19:09:14.532265 ip-10-0-142-116 systemd[1]: Starting Kubernetes Kubelet... Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.986712 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993014 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993037 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993040 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993044 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993047 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993051 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993057 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.055820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993060 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993063 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993066 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993069 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993072 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993075 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993077 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993080 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993083 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993086 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993088 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993093 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993095 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993099 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993102 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993104 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993107 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993110 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993113 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993116 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.059281 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993119 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993122 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993124 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993127 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993130 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993132 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993135 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993138 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993141 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993145 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993150 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993153 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993156 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993158 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993162 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993164 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993167 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993170 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993173 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993175 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.060010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993179 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993182 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993186 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993191 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993196 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993199 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993204 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993207 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993212 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993215 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993217 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993221 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993225 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993228 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993231 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993234 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993236 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993239 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993242 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.062768 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993244 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993247 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993250 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993254 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993257 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993260 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993263 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993266 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993268 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993271 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993274 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993277 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993280 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993283 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993285 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993288 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993290 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993293 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993296 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993299 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.063461 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993302 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993304 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993307 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993312 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993315 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993318 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993321 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993323 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993327 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993330 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993332 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993335 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.993338 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994184 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994193 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994199 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994204 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994207 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994211 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.066267 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994215 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994219 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994222 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994225 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994228 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994231 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994234 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994237 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994240 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994243 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994246 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994249 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994252 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994255 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994257 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994260 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994264 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994268 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994272 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.066949 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994276 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994279 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994281 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994284 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994287 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994289 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994292 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994295 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994299 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994302 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994305 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994309 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994312 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994314 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994317 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994319 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994322 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994325 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994327 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994330 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.069226 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994333 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994335 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994338 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994341 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994346 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994349 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994352 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994354 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994358 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994361 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994364 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994367 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994369 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994372 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994375 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994377 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994380 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994382 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994385 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994388 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.070376 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994390 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994393 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994395 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994398 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994401 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994403 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994406 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994408 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994411 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994413 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994416 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994419 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994421 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994424 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994427 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994430 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994435 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994438 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994441 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994444 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.071358 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994447 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994450 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994454 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994457 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994459 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994462 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994465 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994468 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994471 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994473 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994476 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994479 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994481 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.994484 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995505 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995525 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995531 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995536 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995542 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995547 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.072217 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995552 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995557 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995568 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995572 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995577 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995582 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995587 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995591 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995597 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995602 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995607 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995612 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995619 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995625 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995629 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995640 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995645 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995649 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995654 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.073275 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995660 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995665 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995670 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995675 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995680 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995685 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995689 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995694 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995704 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995712 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995722 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995727 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995732 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995737 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995742 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995747 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995752 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995756 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995761 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.152509 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995766 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.096146 ip-10-0-142-116 systemd[1]: Started Kubernetes Kubelet. Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995776 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995781 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995786 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995791 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995796 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995801 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995805 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995811 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995816 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995821 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995826 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995831 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995841 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995848 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995853 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995858 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995865 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995870 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995876 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.153353 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995884 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995890 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995894 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995899 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995904 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995914 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995919 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995923 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995928 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995933 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995938 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995942 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995947 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995952 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995956 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995961 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995969 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.995980 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996009 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.156520 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996016 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996021 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996025 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996030 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996035 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996040 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996045 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996050 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996056 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996061 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996065 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996075 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996080 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996088 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996093 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.996098 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996286 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996300 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996314 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996320 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996325 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 19:09:15.157302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996329 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996334 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996340 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996344 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996350 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996354 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996358 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996362 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996365 2568 flags.go:64] FLAG: --cgroup-root="" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996369 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996372 2568 flags.go:64] FLAG: --client-ca-file="" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996375 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996379 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996386 2568 flags.go:64] FLAG: --cluster-domain="" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996390 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996393 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996397 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996401 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996406 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996409 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996414 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996420 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996424 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996428 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 19:09:15.158236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996434 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996437 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996441 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996446 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996450 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996453 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996459 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996462 2568 flags.go:64] FLAG: --enable-server="true" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996466 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996475 2568 flags.go:64] FLAG: --event-burst="100" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996479 2568 flags.go:64] FLAG: --event-qps="50" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996483 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996486 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996490 2568 flags.go:64] FLAG: --eviction-hard="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996497 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996500 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996503 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996507 2568 flags.go:64] FLAG: --eviction-soft="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996510 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996513 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996517 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996520 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996523 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996528 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996532 2568 flags.go:64] FLAG: --feature-gates="" Jul 07 19:09:15.159214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996536 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996540 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996544 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996547 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996552 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996555 2568 flags.go:64] FLAG: --help="false" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996561 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996566 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996569 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996573 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996577 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996580 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996584 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996587 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996590 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996596 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996600 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996604 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996607 2568 flags.go:64] FLAG: --kube-reserved="" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996611 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996614 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996617 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996621 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996626 2568 flags.go:64] FLAG: --lock-file="" Jul 07 19:09:15.160034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996630 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996633 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996636 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996643 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996646 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996649 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996652 2568 flags.go:64] FLAG: --logging-format="text" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996656 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996662 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996665 2568 flags.go:64] FLAG: --manifest-url="" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996668 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996674 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996677 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996682 2568 flags.go:64] FLAG: --max-pods="110" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996686 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996691 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996697 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996700 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996704 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996708 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996711 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996721 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996725 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996731 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 19:09:15.161013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996737 2568 flags.go:64] FLAG: --pod-cidr="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996740 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996747 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996750 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996754 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996757 2568 flags.go:64] FLAG: --port="10250" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996760 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996764 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08123af0e5fc2ec34" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996778 2568 flags.go:64] FLAG: --qos-reserved="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996781 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996785 2568 flags.go:64] FLAG: --register-node="true" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996788 2568 flags.go:64] FLAG: --register-with-taints="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996794 2568 flags.go:64] FLAG: --registry-burst="10" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996797 2568 flags.go:64] FLAG: --registry-qps="5" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996801 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996804 2568 flags.go:64] FLAG: --reserved-memory="" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996811 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996815 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996818 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996822 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996825 2568 flags.go:64] FLAG: --runonce="false" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996828 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996832 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996838 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996844 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 19:09:15.161802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996848 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996851 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996855 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996859 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996862 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996865 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996869 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996872 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996879 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996882 2568 flags.go:64] FLAG: --system-cgroups="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996885 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996892 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996896 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996899 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996904 2568 flags.go:64] FLAG: --tls-min-version="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996910 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996913 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996917 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996920 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996924 2568 flags.go:64] FLAG: --v="2" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996928 2568 flags.go:64] FLAG: --version="false" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996933 2568 flags.go:64] FLAG: --vmodule="" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996937 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.996947 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997204 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.288371 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997213 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997217 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997220 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997223 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997226 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997232 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997235 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997238 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997240 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997244 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997247 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997250 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997256 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997259 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997261 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997266 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997269 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997274 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997278 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.289820 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997281 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997284 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997288 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997292 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997296 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997300 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997306 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997309 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997311 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997314 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997317 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997320 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997323 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997326 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997329 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997332 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997335 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997338 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997346 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997349 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.296462 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997352 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997355 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997358 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997361 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997364 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997367 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997369 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997372 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997376 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997379 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997382 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997387 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997390 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997393 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997396 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997399 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997402 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997405 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997408 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997411 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.297352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997413 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997416 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997419 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997425 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997429 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997433 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997437 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997440 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997443 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997445 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997450 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997452 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997456 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997458 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997461 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997467 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997470 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997473 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997476 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.298467 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997479 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997483 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997486 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997489 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997491 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997495 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997498 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997501 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997504 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997509 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997512 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997515 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997518 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997521 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997524 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997526 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997529 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997532 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997535 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.299817 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:14.997537 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:14.997544 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.005477 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.005504 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.005522 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.005528 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005597 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005602 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005606 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005609 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005612 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005615 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005619 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005623 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005626 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.300747 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005629 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005631 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005634 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005637 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005640 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005643 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005646 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005649 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005652 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005655 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005660 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005664 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005667 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005670 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005673 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005676 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005680 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005683 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005687 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005690 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.301620 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005693 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005695 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005698 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005701 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005704 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005707 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005710 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005713 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005716 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005720 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005725 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005728 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005731 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005733 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005736 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005739 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005742 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005745 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.302293 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005749 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005752 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005755 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005758 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005762 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005764 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005767 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005770 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005772 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005775 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005778 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005782 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005785 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005788 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005790 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005793 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005796 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005799 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005802 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005805 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005807 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.303419 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005810 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005812 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005815 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005818 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005820 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005823 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005826 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005828 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005831 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005833 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005836 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005839 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005842 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005844 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005847 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005849 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005852 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005855 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005858 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.304788 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005860 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005863 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005866 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005869 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005872 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005875 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005878 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005880 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005887 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005890 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005893 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.005896 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.005901 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006049 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:09:15.305412 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006054 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006058 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006062 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006065 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006068 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006071 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006075 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006078 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006081 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006085 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006088 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006091 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006094 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006096 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006099 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006102 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006105 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006107 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006110 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:09:15.305811 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006113 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006119 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006122 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006125 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006129 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006131 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006134 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006138 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006141 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006143 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006146 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006149 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006151 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006154 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006157 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006160 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006162 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006165 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006168 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006171 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:09:15.306335 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006173 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006176 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006179 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006181 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006184 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006187 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006189 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006192 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006195 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006197 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006200 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006202 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006205 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006209 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006212 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006215 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006218 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006221 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006224 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:09:15.306894 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006229 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006233 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006236 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006239 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006242 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006245 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006248 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006251 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006254 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006257 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006260 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006263 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006266 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006269 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006272 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006275 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006277 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006280 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006282 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006285 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:09:15.307435 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006288 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006290 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006293 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006295 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006298 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006301 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006304 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006307 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006310 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006312 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006315 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006319 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006322 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006325 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006328 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006331 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006333 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006336 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006339 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:09:15.307975 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:15.006341 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.006347 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.006492 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.009096 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.010566 2568 server.go:1019] "Starting client certificate rotation" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.010681 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.010770 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.014137 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.015409 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.039335 2568 log.go:25] "Validated CRI v1 runtime API" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.042003 2568 log.go:25] "Validated CRI v1 image API" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.043506 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 19:09:15.308508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.050252 2568 fs.go:135] Filesystem UUIDs: map[0ffc2149-b621-41f9-a4c6-47e0bb1a3601:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d6a8f66d-71e5-44eb-8d45-6a5047208d8d:/dev/nvme0n1p3] Jul 07 19:09:15.308890 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.050275 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.056826 2568 manager.go:216] Machine: {Timestamp:2026-07-07 19:09:15.055378256 +0000 UTC m=+0.406026506 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200580 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2321cf6b9b9571c5b1c71b076fd2dd SystemUUID:ec2321cf-6b9b-9571-c5b1-c71b076fd2dd BootID:58937ddf-0d85-4f37-b522-dd0f00694b89 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ea:4d:f0:75:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ea:4d:f0:75:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:d9:a6:03:82:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.057020 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.057166 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.058479 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.058522 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.060724 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.060748 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 19:09:15.308923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.060759 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.061438 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.063164 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.066979 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.067207 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068056 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068084 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068107 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068160 2568 kubelet.go:398] "Adding apiserver pod source" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068179 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068956 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.068996 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.072075 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.074372 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.074453 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89cps" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.074497 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.074533 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076496 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076513 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076520 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076526 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076536 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076542 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076551 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076559 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076569 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076576 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076585 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.076590 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.078642 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.078655 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.080262 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.080307 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.084507 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89cps" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.095028 2568 server.go:1268] "Started kubelet" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.095164 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.095182 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.095240 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.097366 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.098168 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 07 19:09:15.309204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.104484 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.104496 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.104638 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.105272 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.105292 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.105271 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.105384 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.105391 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.109862 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110017 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110031 2568 factory.go:55] Registering systemd factory Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110042 2568 factory.go:223] Registration of the systemd container factory successfully Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110335 2568 factory.go:153] Registering CRI-O factory Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110350 2568 factory.go:223] Registration of the crio container factory successfully Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110379 2568 factory.go:103] Registering Raw factory Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110395 2568 manager.go:1196] Started watching for new ooms in manager Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.110550 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.110995 2568 manager.go:349] Starting recovery of all containers Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.113381 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-116.ec2.internal" not found Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.113457 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-116.ec2.internal\" not found" node="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.121693 2568 manager.go:354] Recovery completed Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.127575 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.129900 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.129943 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.129956 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.130657 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.130667 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.130687 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.130670 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.131511 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.131538 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.131548 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.131673 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-116.ec2.internal" not found Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.132444 2568 policy_none.go:49] "None policy: Start" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.132458 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.132469 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.134761 2568 policy_none.go:47] "Start" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.193268 2568 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-116.ec2.internal" not found Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197418 2568 manager.go:337] "Starting Device Plugin manager" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.197451 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197461 2568 server.go:87] "Starting device plugin registration server" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197772 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197786 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 19:09:15.310313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197899 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.197997 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.198006 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.198598 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.198663 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.298929 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.300306 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.300348 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.300362 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.300402 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.308959 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.311484 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.309014 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-116.ec2.internal\": node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.328671 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.328636 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.344215 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.344186 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 19:09:15.345521 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.345503 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 19:09:15.345573 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.345563 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 19:09:15.345641 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.345594 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 19:09:15.345676 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.345637 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 19:09:15.353170 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.353120 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:15.429997 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.429955 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.446340 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.446298 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal"] Jul 07 19:09:15.446444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.446400 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.448168 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.448143 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.448275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.448191 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.448275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.448205 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.449801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.449786 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.449920 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.449903 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.449981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.449969 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.450788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450772 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.450880 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450800 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.450880 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450810 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.450880 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450773 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.451011 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450885 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.451011 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.450896 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.452326 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.452303 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.452425 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.452335 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:09:15.453175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.453159 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:09:15.453267 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.453194 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:09:15.453267 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.453209 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:09:15.481179 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.481153 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-116.ec2.internal\" not found" node="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.486200 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.486180 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-116.ec2.internal\" not found" node="ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.508513 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.508483 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85dca3ea545cba1033034b0f993d4dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.ec2.internal\" (UID: \"85dca3ea545cba1033034b0f993d4dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.508634 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.508520 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.508634 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.508543 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.530103 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.530033 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.608689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608652 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.608689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.608873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85dca3ea545cba1033034b0f993d4dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.ec2.internal\" (UID: \"85dca3ea545cba1033034b0f993d4dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.608873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.608873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608756 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7f157e784775de6711a5b9059e36d88-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal\" (UID: \"f7f157e784775de6711a5b9059e36d88\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.608873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.608784 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/85dca3ea545cba1033034b0f993d4dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.ec2.internal\" (UID: \"85dca3ea545cba1033034b0f993d4dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.630134 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.630108 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.731388 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.731349 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.787885 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.787810 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.789026 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:15.789008 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:15.832497 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.832464 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:15.934019 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:15.933973 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:16.009466 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.009427 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 19:09:16.009670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.009646 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:09:16.009721 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.009647 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:09:16.034703 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:16.034663 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:16.089576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.089449 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 19:04:15 +0000 UTC" deadline="2027-12-22 06:03:51.847043272 +0000 UTC" Jul 07 19:09:16.089576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.089511 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12778h54m35.757536206s" Jul 07 19:09:16.135692 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:16.135657 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-116.ec2.internal\" not found" Jul 07 19:09:16.171985 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.171957 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:16.205838 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.205809 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" Jul 07 19:09:16.223413 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.223390 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:09:16.225395 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.225378 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" Jul 07 19:09:16.237310 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.237283 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:09:16.269689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.269659 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:16.605331 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:16.605286 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dca3ea545cba1033034b0f993d4dd5.slice/crio-9a658336f6a861accd1d8a6aedcd016b064587e5fef487cff7be410e3495a668 WatchSource:0}: Error finding container 9a658336f6a861accd1d8a6aedcd016b064587e5fef487cff7be410e3495a668: Status 404 returned error can't find the container with id 9a658336f6a861accd1d8a6aedcd016b064587e5fef487cff7be410e3495a668 Jul 07 19:09:16.611062 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:16.611042 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:09:17.070084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.070052 2568 apiserver.go:52] "Watching apiserver" Jul 07 19:09:17.076140 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.076114 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 19:09:17.079002 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.078963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-g4dct","openshift-cluster-node-tuning-operator/tuned-nkmj5","openshift-dns/node-resolver-2p956","openshift-image-registry/node-ca-xhfcg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal","openshift-multus/multus-6xb9v","openshift-network-diagnostics/network-check-target-j796b","openshift-ovn-kubernetes/ovnkube-node-4z7b4","kube-system/konnectivity-agent-h7g62","kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp","openshift-multus/multus-additional-cni-plugins-dnk94","openshift-multus/network-metrics-daemon-lk246","openshift-network-operator/iptables-alerter-2ft72"] Jul 07 19:09:17.081231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.081214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:17.081304 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.081284 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:17.082351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.082331 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.084017 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.083999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.084537 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.084524 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.084745 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.084733 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 19:09:17.084871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.084856 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7pgc4\"" Jul 07 19:09:17.084926 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.084917 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.085305 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.085293 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.086088 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.086071 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.086184 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.086075 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-plnz9\"" Jul 07 19:09:17.086622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.086604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.086884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.086867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.087501 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.087479 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.087841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.087823 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.087948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.087847 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:17.087948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.087855 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5tg95\"" Jul 07 19:09:17.088288 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.088269 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 19:09:17.088374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.088314 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.089742 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.089719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.091009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.090979 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.091082 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.091036 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:17.092324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092070 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.092324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092101 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 19:09:17.092324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092205 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.092324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092252 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.092519 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092502 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.092558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092545 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 19:09:17.092590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092565 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qznx5\"" Jul 07 19:09:17.093165 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.092989 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 19:09:17.093509 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.093315 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.093509 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.093336 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xd98w\"" Jul 07 19:09:17.093697 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.093678 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 19:09:17.094278 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.094257 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 19:09:17.094645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.094354 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nqknm\"" Jul 07 19:09:17.094794 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.094777 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.096113 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.096086 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.096212 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.096193 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mz47m\"" Jul 07 19:09:17.096864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.096448 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.096864 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.096559 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:17.097515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097497 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.097646 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097625 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 19:09:17.097706 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097651 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 19:09:17.097706 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097667 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 19:09:17.097706 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097689 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 19:09:17.097824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lcxr6\"" Jul 07 19:09:17.097824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097548 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 19:09:17.098119 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.097984 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 19:09:17.098224 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.098203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.101533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.101509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 19:09:17.101644 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.101539 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 19:09:17.101869 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.101856 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-59nvl\"" Jul 07 19:09:17.106349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.106335 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 19:09:17.106422 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.106397 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 19:09:17.119229 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zn8h\" (UniqueName: \"kubernetes.io/projected/18c0791b-25b2-4c9f-afa4-b49206b43b7a-kube-api-access-5zn8h\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.119333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-node-log\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.119333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119268 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-host\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.119333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-etc-tuned\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.119440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119340 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-system-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.119440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-env-overrides\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.119440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-registration-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-dbus\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e39206e-6b9b-4a92-a0ed-705902137faf-hosts-file\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cnibin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-lib-modules\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.119560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-conf-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-var-lib-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119605 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-netd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtkw\" (UniqueName: \"kubernetes.io/projected/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-kube-api-access-xxtkw\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysconfig\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119669 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ds2\" (UniqueName: \"kubernetes.io/projected/60f25131-9545-4778-809d-718358980a34-kube-api-access-x7ds2\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:17.119760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-socket-dir-parent\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-kubelet\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-systemd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-iptables-alerter-script\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1535d4c-d58e-4fcc-addd-65cd866a41bd-agent-certs\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-ovn\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119967 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-k8s-cni-cncf-io\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.119994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-netns\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-systemd\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghpv\" (UniqueName: \"kubernetes.io/projected/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-kube-api-access-qghpv\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-kubelet\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-hostroot\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-etc-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.120257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-etc-selinux\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1535d4c-d58e-4fcc-addd-65cd866a41bd-konnectivity-ca\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-os-release\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cni-binary-copy\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-multus-certs\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-etc-kubernetes\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-config\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120450 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-device-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-sys-fs\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-netns\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-script-lib\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-kubelet-config\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csd8v\" (UniqueName: \"kubernetes.io/projected/2dfb8567-19e4-48fa-98a5-68e65b8a3378-kube-api-access-csd8v\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-slash\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-kubernetes\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-os-release\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqk8\" (UniqueName: \"kubernetes.io/projected/8120d1f0-2e6a-40e7-8017-7923fce34250-kube-api-access-mmqk8\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39206e-6b9b-4a92-a0ed-705902137faf-tmp-dir\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6j6w\" (UniqueName: \"kubernetes.io/projected/6e39206e-6b9b-4a92-a0ed-705902137faf-kube-api-access-d6j6w\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-multus\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-log-socket\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.120930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfb8567-19e4-48fa-98a5-68e65b8a3378-host\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-bin\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120956 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssqq\" (UniqueName: \"kubernetes.io/projected/9b06a6ce-f447-4e16-974f-48c31d448a5a-kube-api-access-fssqq\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-conf\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.120997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-sys\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovn-node-metrics-cert\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121079 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfb8567-19e4-48fa-98a5-68e65b8a3378-serviceca\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-daemon-config\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-systemd-units\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-modprobe-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cnibin\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121222 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-bin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.121536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-host-slash\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-system-cni-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-socket-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fsb9\" (UniqueName: \"kubernetes.io/projected/8c996d6e-041b-4ace-9a6e-898b26328689-kube-api-access-7fsb9\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-run\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-var-lib-kubelet\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.121981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.121510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-tmp\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.131433 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.131412 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:09:17.162450 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.162421 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-72r2p" Jul 07 19:09:17.176298 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.176241 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-72r2p" Jul 07 19:09:17.222276 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-system-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.222276 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-env-overrides\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-registration-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-dbus\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e39206e-6b9b-4a92-a0ed-705902137faf-hosts-file\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cnibin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222404 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-lib-modules\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-conf-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222452 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-var-lib-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-netd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtkw\" (UniqueName: \"kubernetes.io/projected/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-kube-api-access-xxtkw\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.222536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222523 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysconfig\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ds2\" (UniqueName: \"kubernetes.io/projected/60f25131-9545-4778-809d-718358980a34-kube-api-access-x7ds2\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-socket-dir-parent\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222635 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-kubelet\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-systemd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222695 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-iptables-alerter-script\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1535d4c-d58e-4fcc-addd-65cd866a41bd-agent-certs\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222772 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-system-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-ovn\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-k8s-cni-cncf-io\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-netns\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-systemd\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghpv\" (UniqueName: \"kubernetes.io/projected/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-kube-api-access-qghpv\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-kubelet\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-hostroot\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.222978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-etc-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-etc-selinux\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1535d4c-d58e-4fcc-addd-65cd866a41bd-konnectivity-ca\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-os-release\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cni-binary-copy\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-multus-certs\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-etc-kubernetes\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.223730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-config\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-device-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-sys-fs\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-netns\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223304 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e39206e-6b9b-4a92-a0ed-705902137faf-hosts-file\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-env-overrides\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223320 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-dbus\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-script-lib\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223394 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cnibin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-kubelet-config\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223442 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/134d60c9-28cc-4320-ac6b-740976d6382d-kubelet-config\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223450 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223528 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.224408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csd8v\" (UniqueName: \"kubernetes.io/projected/2dfb8567-19e4-48fa-98a5-68e65b8a3378-kube-api-access-csd8v\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-slash\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-kubernetes\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-os-release\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqk8\" (UniqueName: \"kubernetes.io/projected/8120d1f0-2e6a-40e7-8017-7923fce34250-kube-api-access-mmqk8\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39206e-6b9b-4a92-a0ed-705902137faf-tmp-dir\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223745 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-lib-modules\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.223781 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223793 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-conf-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-script-lib\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223835 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-var-lib-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223880 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-netd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223892 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-socket-dir-parent\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.223948 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:17.723927398 +0000 UTC m=+3.074575620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:17.224827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223954 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-multus-certs\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223970 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-os-release\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223982 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39206e-6b9b-4a92-a0ed-705902137faf-tmp-dir\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224023 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-etc-kubernetes\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223992 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-etc-selinux\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224080 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-ovn\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224092 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-device-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224148 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-sys-fs\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224187 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-netns\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-systemd\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224247 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224270 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-kubelet\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysconfig\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224317 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-hostroot\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224361 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-k8s-cni-cncf-io\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.225339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224404 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-run-netns\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224456 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-systemd\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224528 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1535d4c-d58e-4fcc-addd-65cd866a41bd-konnectivity-ca\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6j6w\" (UniqueName: \"kubernetes.io/projected/6e39206e-6b9b-4a92-a0ed-705902137faf-kube-api-access-d6j6w\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovnkube-config\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-multus\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224601 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-registration-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224618 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-cni-binary-copy\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-log-socket\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224609 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224659 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-log-socket\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfb8567-19e4-48fa-98a5-68e65b8a3378-host\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224712 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-kubelet\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-bin\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.223548 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssqq\" (UniqueName: \"kubernetes.io/projected/9b06a6ce-f447-4e16-974f-48c31d448a5a-kube-api-access-fssqq\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-conf\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224852 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-etc-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224857 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-kubernetes\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-sys\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dfb8567-19e4-48fa-98a5-68e65b8a3378-host\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovn-node-metrics-cert\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-multus\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224953 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-os-release\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.224987 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-iptables-alerter-script\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.225046 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225065 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-slash\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.226923 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225077 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-sysctl-conf\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.225094 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:17.725077639 +0000 UTC m=+3.075725855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-host-cni-bin\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfb8567-19e4-48fa-98a5-68e65b8a3378-serviceca\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225134 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-sys\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225173 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-daemon-config\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-systemd-units\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-modprobe-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cnibin\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225369 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cnibin\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-bin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-etc-modprobe-d\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.227657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-host-slash\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-system-cni-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225499 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-systemd-units\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225542 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-host-var-lib-cni-bin\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225543 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-system-cni-dir\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225565 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2dfb8567-19e4-48fa-98a5-68e65b8a3378-serviceca\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225600 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-cni-dir\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-socket-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225703 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-run-openvswitch\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsb9\" (UniqueName: \"kubernetes.io/projected/8c996d6e-041b-4ace-9a6e-898b26328689-kube-api-access-7fsb9\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-run\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c996d6e-041b-4ace-9a6e-898b26328689-socket-dir\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.228427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-var-lib-kubelet\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-tmp\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225868 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-var-lib-kubelet\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225894 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-host-slash\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225948 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-run\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.225993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn8h\" (UniqueName: \"kubernetes.io/projected/18c0791b-25b2-4c9f-afa4-b49206b43b7a-kube-api-access-5zn8h\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-node-log\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226020 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-host\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-etc-tuned\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226295 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/60f25131-9545-4778-809d-718358980a34-host\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226295 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b06a6ce-f447-4e16-974f-48c31d448a5a-node-log\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.226755 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-cni-binary-copy\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.229143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.228980 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1535d4c-d58e-4fcc-addd-65cd866a41bd-agent-certs\") pod \"konnectivity-agent-h7g62\" (UID: \"c1535d4c-d58e-4fcc-addd-65cd866a41bd\") " pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.229791 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.229204 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-tmp\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.229791 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.229525 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/60f25131-9545-4778-809d-718358980a34-etc-tuned\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.230268 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.230242 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18c0791b-25b2-4c9f-afa4-b49206b43b7a-multus-daemon-config\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.230475 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.230450 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b06a6ce-f447-4e16-974f-48c31d448a5a-ovn-node-metrics-cert\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.234053 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.233968 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csd8v\" (UniqueName: \"kubernetes.io/projected/2dfb8567-19e4-48fa-98a5-68e65b8a3378-kube-api-access-csd8v\") pod \"node-ca-xhfcg\" (UID: \"2dfb8567-19e4-48fa-98a5-68e65b8a3378\") " pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.238077 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.238053 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:17.238077 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.238079 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:17.238270 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.238093 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:17.238270 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.238171 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:17.738151034 +0000 UTC m=+3.088799251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:17.240879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.240853 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ds2\" (UniqueName: \"kubernetes.io/projected/60f25131-9545-4778-809d-718358980a34-kube-api-access-x7ds2\") pod \"tuned-nkmj5\" (UID: \"60f25131-9545-4778-809d-718358980a34\") " pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.241082 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.241063 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtkw\" (UniqueName: \"kubernetes.io/projected/7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe-kube-api-access-xxtkw\") pod \"iptables-alerter-2ft72\" (UID: \"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe\") " pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.246444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.246413 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqk8\" (UniqueName: \"kubernetes.io/projected/8120d1f0-2e6a-40e7-8017-7923fce34250-kube-api-access-mmqk8\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.246553 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.246498 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsb9\" (UniqueName: \"kubernetes.io/projected/8c996d6e-041b-4ace-9a6e-898b26328689-kube-api-access-7fsb9\") pod \"aws-ebs-csi-driver-node-7w2dp\" (UID: \"8c996d6e-041b-4ace-9a6e-898b26328689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.246869 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.246849 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6j6w\" (UniqueName: \"kubernetes.io/projected/6e39206e-6b9b-4a92-a0ed-705902137faf-kube-api-access-d6j6w\") pod \"node-resolver-2p956\" (UID: \"6e39206e-6b9b-4a92-a0ed-705902137faf\") " pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.247369 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.247081 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghpv\" (UniqueName: \"kubernetes.io/projected/dea3f7e8-5e7b-400f-9eef-551398fe4ad4-kube-api-access-qghpv\") pod \"multus-additional-cni-plugins-dnk94\" (UID: \"dea3f7e8-5e7b-400f-9eef-551398fe4ad4\") " pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.248140 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.248112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn8h\" (UniqueName: \"kubernetes.io/projected/18c0791b-25b2-4c9f-afa4-b49206b43b7a-kube-api-access-5zn8h\") pod \"multus-6xb9v\" (UID: \"18c0791b-25b2-4c9f-afa4-b49206b43b7a\") " pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.248499 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.248478 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssqq\" (UniqueName: \"kubernetes.io/projected/9b06a6ce-f447-4e16-974f-48c31d448a5a-kube-api-access-fssqq\") pod \"ovnkube-node-4z7b4\" (UID: \"9b06a6ce-f447-4e16-974f-48c31d448a5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.350546 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.350426 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" event={"ID":"85dca3ea545cba1033034b0f993d4dd5","Type":"ContainerStarted","Data":"9a658336f6a861accd1d8a6aedcd016b064587e5fef487cff7be410e3495a668"} Jul 07 19:09:17.393165 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.393123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" Jul 07 19:09:17.401549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.401515 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2p956" Jul 07 19:09:17.405724 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.405684 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c996d6e_041b_4ace_9a6e_898b26328689.slice/crio-ccb50b68311cced04a83636405537c4ec2b01900b827936a75f8aac3079449ce WatchSource:0}: Error finding container ccb50b68311cced04a83636405537c4ec2b01900b827936a75f8aac3079449ce: Status 404 returned error can't find the container with id ccb50b68311cced04a83636405537c4ec2b01900b827936a75f8aac3079449ce Jul 07 19:09:17.408513 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.408487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xhfcg" Jul 07 19:09:17.411352 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.411326 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e39206e_6b9b_4a92_a0ed_705902137faf.slice/crio-3289cd0ecbaba1fc6d43739cf818542b829beaba859127f67a3efa9b3a4ab82e WatchSource:0}: Error finding container 3289cd0ecbaba1fc6d43739cf818542b829beaba859127f67a3efa9b3a4ab82e: Status 404 returned error can't find the container with id 3289cd0ecbaba1fc6d43739cf818542b829beaba859127f67a3efa9b3a4ab82e Jul 07 19:09:17.414003 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.413975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2ft72" Jul 07 19:09:17.418778 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.418746 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb8567_19e4_48fa_98a5_68e65b8a3378.slice/crio-57220afbf02bb87e81bb09feee00fc90acffc839f0b814c74e551fc08089eb17 WatchSource:0}: Error finding container 57220afbf02bb87e81bb09feee00fc90acffc839f0b814c74e551fc08089eb17: Status 404 returned error can't find the container with id 57220afbf02bb87e81bb09feee00fc90acffc839f0b814c74e551fc08089eb17 Jul 07 19:09:17.420200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.420174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xb9v" Jul 07 19:09:17.426154 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.425779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:17.426154 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.425982 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2b0dc2_ab9c_478e_b0e9_2aa88a619ffe.slice/crio-e0baf59163199b298647f04ebd1ddc19a53a652dbba9daa7da6910776ed68574 WatchSource:0}: Error finding container e0baf59163199b298647f04ebd1ddc19a53a652dbba9daa7da6910776ed68574: Status 404 returned error can't find the container with id e0baf59163199b298647f04ebd1ddc19a53a652dbba9daa7da6910776ed68574 Jul 07 19:09:17.432216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.432193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" Jul 07 19:09:17.437804 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.437131 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c0791b_25b2_4c9f_afa4_b49206b43b7a.slice/crio-e10a36afa6e587779955950a8116e3c5e56e1471fa12d99dd5647886017d91a5 WatchSource:0}: Error finding container e10a36afa6e587779955950a8116e3c5e56e1471fa12d99dd5647886017d91a5: Status 404 returned error can't find the container with id e10a36afa6e587779955950a8116e3c5e56e1471fa12d99dd5647886017d91a5 Jul 07 19:09:17.438161 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.438133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:17.438826 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.438802 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1535d4c_d58e_4fcc_addd_65cd866a41bd.slice/crio-1e1af3cfbba706744ae5bbb6fb08ad9840c31289504ff10eee9adcbbc7b80f69 WatchSource:0}: Error finding container 1e1af3cfbba706744ae5bbb6fb08ad9840c31289504ff10eee9adcbbc7b80f69: Status 404 returned error can't find the container with id 1e1af3cfbba706744ae5bbb6fb08ad9840c31289504ff10eee9adcbbc7b80f69 Jul 07 19:09:17.442631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.442611 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dnk94" Jul 07 19:09:17.445505 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.445476 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f25131_9545_4778_809d_718358980a34.slice/crio-423f61c88d7035d0d228988ab3329a8bf4393f586833b77713753ecf9399756a WatchSource:0}: Error finding container 423f61c88d7035d0d228988ab3329a8bf4393f586833b77713753ecf9399756a: Status 404 returned error can't find the container with id 423f61c88d7035d0d228988ab3329a8bf4393f586833b77713753ecf9399756a Jul 07 19:09:17.455031 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:17.454968 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea3f7e8_5e7b_400f_9eef_551398fe4ad4.slice/crio-240292e5082954b13981e4dea86672f7b76e8d174278c92e9e99485d9b0fdc1b WatchSource:0}: Error finding container 240292e5082954b13981e4dea86672f7b76e8d174278c92e9e99485d9b0fdc1b: Status 404 returned error can't find the container with id 240292e5082954b13981e4dea86672f7b76e8d174278c92e9e99485d9b0fdc1b Jul 07 19:09:17.510881 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.510845 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:09:17.731249 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.731214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:17.731428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.731265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:17.731428 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.731400 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:17.731556 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.731400 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:17.731556 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.731459 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:18.731444151 +0000 UTC m=+4.082092367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:17.731556 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.731527 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:18.731506696 +0000 UTC m=+4.082154911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:17.832573 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:17.832522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:17.832765 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.832742 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:17.832846 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.832775 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:17.832846 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.832792 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:17.832928 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:17.832864 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:18.832842998 +0000 UTC m=+4.183491215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:18.178968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.178879 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:04:17 +0000 UTC" deadline="2028-03-20 12:00:56.034601084 +0000 UTC" Jul 07 19:09:18.178968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.178960 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14920h51m37.855645605s" Jul 07 19:09:18.353735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.353693 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"74be4ccf32a760c098818e356eede31c9b3c04008a94b717c9f96d22de7fce97"} Jul 07 19:09:18.355198 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.355164 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" event={"ID":"60f25131-9545-4778-809d-718358980a34","Type":"ContainerStarted","Data":"423f61c88d7035d0d228988ab3329a8bf4393f586833b77713753ecf9399756a"} Jul 07 19:09:18.356465 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.356436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xb9v" event={"ID":"18c0791b-25b2-4c9f-afa4-b49206b43b7a","Type":"ContainerStarted","Data":"e10a36afa6e587779955950a8116e3c5e56e1471fa12d99dd5647886017d91a5"} Jul 07 19:09:18.357819 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.357789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2ft72" event={"ID":"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe","Type":"ContainerStarted","Data":"e0baf59163199b298647f04ebd1ddc19a53a652dbba9daa7da6910776ed68574"} Jul 07 19:09:18.359109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.359065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xhfcg" event={"ID":"2dfb8567-19e4-48fa-98a5-68e65b8a3378","Type":"ContainerStarted","Data":"57220afbf02bb87e81bb09feee00fc90acffc839f0b814c74e551fc08089eb17"} Jul 07 19:09:18.360393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.360368 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2p956" event={"ID":"6e39206e-6b9b-4a92-a0ed-705902137faf","Type":"ContainerStarted","Data":"3289cd0ecbaba1fc6d43739cf818542b829beaba859127f67a3efa9b3a4ab82e"} Jul 07 19:09:18.361785 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.361742 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerStarted","Data":"240292e5082954b13981e4dea86672f7b76e8d174278c92e9e99485d9b0fdc1b"} Jul 07 19:09:18.363062 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.363038 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h7g62" event={"ID":"c1535d4c-d58e-4fcc-addd-65cd866a41bd","Type":"ContainerStarted","Data":"1e1af3cfbba706744ae5bbb6fb08ad9840c31289504ff10eee9adcbbc7b80f69"} Jul 07 19:09:18.364240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.364216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" event={"ID":"8c996d6e-041b-4ace-9a6e-898b26328689","Type":"ContainerStarted","Data":"ccb50b68311cced04a83636405537c4ec2b01900b827936a75f8aac3079449ce"} Jul 07 19:09:18.740542 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.740164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:18.740542 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.740447 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:18.740542 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.740498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:18.740853 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.740587 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.740564179 +0000 UTC m=+6.091212399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:18.740853 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.740630 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:18.740853 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.740684 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.740668209 +0000 UTC m=+6.091316431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:18.841731 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:18.841687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:18.841962 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.841937 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:18.842058 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.841979 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:18.842058 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.841995 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:18.842180 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:18.842061 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.842038675 +0000 UTC m=+6.192686894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:19.179538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.179466 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:04:17 +0000 UTC" deadline="2027-12-17 15:34:43.518969089 +0000 UTC" Jul 07 19:09:19.179538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.179506 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12668h25m24.33946814s" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.346685 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:19.346847 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.347079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:19.347188 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.347353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:19.347876 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:19.347442 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:19.395052 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:19.394971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" event={"ID":"85dca3ea545cba1033034b0f993d4dd5","Type":"ContainerStarted","Data":"f7b20bd27c8725ec368b7710fc995bcb0e496c9a24e6099fd8055916eba63e52"} Jul 07 19:09:20.760954 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:20.760890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:20.761439 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:20.760979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:20.761439 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.761137 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:20.761439 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.761202 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:24.761183298 +0000 UTC m=+10.111831525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:20.761697 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.761644 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:20.761765 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.761699 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:24.761683972 +0000 UTC m=+10.112332193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:20.862054 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:20.861980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:20.862242 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.862212 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:20.862302 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.862231 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:20.862302 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.862292 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:20.862397 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:20.862351 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:24.862333741 +0000 UTC m=+10.212981958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:21.346436 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:21.346383 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:21.346611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:21.346383 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:21.346611 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:21.346527 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:21.346611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:21.346538 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:21.346790 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:21.346631 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:21.346790 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:21.346714 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:23.346940 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:23.346884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:23.346940 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:23.346909 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:23.346940 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:23.346943 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:23.347512 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:23.347048 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:23.347512 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:23.347187 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:23.347512 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:23.347321 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:24.794251 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:24.794198 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:24.794738 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:24.794301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:24.794738 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.794381 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:24.794738 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.794404 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:24.794738 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.794468 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:32.794445517 +0000 UTC m=+18.145093744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:24.794738 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.794490 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:32.794479386 +0000 UTC m=+18.145127628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:24.895001 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:24.894965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:24.895188 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.895108 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:24.895188 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.895129 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:24.895188 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.895141 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:24.895336 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:24.895203 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:32.895186021 +0000 UTC m=+18.245834249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:25.347375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:25.347529 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:25.347908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:25.348027 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:25.348085 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:25.348189 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:25.348154 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:27.346945 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:27.346669 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:27.346945 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:27.346695 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:27.346945 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:27.346665 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:27.346945 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:27.346798 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:27.346945 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:27.346859 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:27.347598 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:27.346972 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:28.420530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:28.420489 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" event={"ID":"60f25131-9545-4778-809d-718358980a34","Type":"ContainerStarted","Data":"142e406377d2d318726d1ddb5636895d6cd840bdfb726a82055864eafc6a986d"} Jul 07 19:09:28.424661 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:28.424621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" event={"ID":"f7f157e784775de6711a5b9059e36d88","Type":"ContainerStarted","Data":"4df299e86a6eed17a0d05dd4598f33cfc6189832d8dfd69baf5bd96b65f0e5d0"} Jul 07 19:09:28.442428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:28.442332 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.ec2.internal" podStartSLOduration=12.442310633 podStartE2EDuration="12.442310633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:19.427778199 +0000 UTC m=+4.778426438" watchObservedRunningTime="2026-07-07 19:09:28.442310633 +0000 UTC m=+13.792958873" Jul 07 19:09:29.346263 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.345995 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:29.346467 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:29.346382 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:29.346467 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.346118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:29.346582 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:29.346482 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:29.346582 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.346064 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:29.346582 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:29.346544 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:29.428547 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.428502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xhfcg" event={"ID":"2dfb8567-19e4-48fa-98a5-68e65b8a3378","Type":"ContainerStarted","Data":"e8bfcb808264f807bd53d3488da60362604e38fce4db742f0ba3a77c27b69ad1"} Jul 07 19:09:29.430683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.430648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2p956" event={"ID":"6e39206e-6b9b-4a92-a0ed-705902137faf","Type":"ContainerStarted","Data":"12a97f4786625426a092cc5023f7f20e2460d9beb08615aa3efe88c979c34424"} Jul 07 19:09:29.432810 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.432777 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerStarted","Data":"57bc8832dff46247af91e7c617c08bad301ad626d1c36d1bb59fdd2674e226e4"} Jul 07 19:09:29.434620 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.434584 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h7g62" event={"ID":"c1535d4c-d58e-4fcc-addd-65cd866a41bd","Type":"ContainerStarted","Data":"493e6d073acfb146bba0935bbdb99d3afdbf2e6aaa7d45098a0aae688b3efaaa"} Jul 07 19:09:29.436997 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.436966 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" event={"ID":"8c996d6e-041b-4ace-9a6e-898b26328689","Type":"ContainerStarted","Data":"d274e79583ffd08d12f5c3a5443cf9c48f45aa3a7b83ee0f56ccbe0479111ec3"} Jul 07 19:09:29.447910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.447848 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nkmj5" podStartSLOduration=4.157427445 podStartE2EDuration="14.44782815s" totalImagesPullingTime="10.290400705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:28.441699984 +0000 UTC m=+13.792348246" watchObservedRunningTime="2026-07-07 19:09:29.44782815 +0000 UTC m=+14.798476378" Jul 07 19:09:29.448125 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.448071 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xhfcg" podStartSLOduration=4.15519614 podStartE2EDuration="14.4480624s" totalImagesPullingTime="10.29286626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:29.447257132 +0000 UTC m=+14.797905394" watchObservedRunningTime="2026-07-07 19:09:29.4480624 +0000 UTC m=+14.798710640" Jul 07 19:09:29.497423 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:29.497373 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h7g62" podStartSLOduration=4.2155217 podStartE2EDuration="14.497341873s" totalImagesPullingTime="10.281820173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:29.468789353 +0000 UTC m=+14.819437592" watchObservedRunningTime="2026-07-07 19:09:29.497341873 +0000 UTC m=+14.847990113" Jul 07 19:09:30.440976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:30.440910 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2ft72" event={"ID":"7e2b0dc2-ab9c-478e-b0e9-2aa88a619ffe","Type":"ContainerStarted","Data":"bc4d884d3637e4acca0d887f28a3766369eb1b2d06822a2a975f75e8dd6f019a"} Jul 07 19:09:30.458144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:30.458094 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2ft72" podStartSLOduration=5.248377357 podStartE2EDuration="15.458077171s" totalImagesPullingTime="10.209699814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:30.457900649 +0000 UTC m=+15.808548887" watchObservedRunningTime="2026-07-07 19:09:30.458077171 +0000 UTC m=+15.808725407" Jul 07 19:09:30.458368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:30.458337 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2p956" podStartSLOduration=5.150541191 podStartE2EDuration="15.458321688s" totalImagesPullingTime="10.307780497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:29.517733038 +0000 UTC m=+14.868381276" watchObservedRunningTime="2026-07-07 19:09:30.458321688 +0000 UTC m=+15.808969925" Jul 07 19:09:31.345934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:31.345893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:31.346120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:31.345893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:31.346120 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:31.346035 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:31.346120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:31.345891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:31.346120 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:31.346098 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:31.346262 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:31.346180 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:31.545565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:31.545521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:31.546260 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:31.546234 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:32.857714 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:32.857671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:32.858381 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:32.857767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:32.858381 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.857868 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:32.858381 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.857875 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:32.858381 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.857957 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:48.857937935 +0000 UTC m=+34.208586155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:32.858381 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.858002 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:48.857984058 +0000 UTC m=+34.208632288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:32.958483 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:32.958428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:32.958693 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.958646 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:32.958693 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.958667 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:32.958693 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.958680 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:32.958850 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:32.958740 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:09:48.95872066 +0000 UTC m=+34.309368879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:33.346072 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:33.346034 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:33.346072 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:33.346062 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:33.346300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:33.346034 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:33.346300 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:33.346174 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:33.346300 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:33.346249 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:33.346452 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:33.346364 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:35.349208 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:35.349124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:35.349823 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:35.349566 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:35.350266 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:35.350245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:35.350424 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:35.350403 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:35.350534 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:35.350521 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:35.350670 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:35.350651 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:37.346727 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.346684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:37.347243 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.346739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:37.347243 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.346684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:37.347243 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:37.346853 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:37.347243 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:37.346946 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:37.347243 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:37.347039 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:37.462497 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.462041 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="57bc8832dff46247af91e7c617c08bad301ad626d1c36d1bb59fdd2674e226e4" exitCode=0 Jul 07 19:09:37.462497 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.462200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"57bc8832dff46247af91e7c617c08bad301ad626d1c36d1bb59fdd2674e226e4"} Jul 07 19:09:37.598898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:37.598873 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 19:09:38.209825 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.209499 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:09:37.598890279Z","UUID":"18eefd14-8dc8-43d1-b1d1-6ea90bdc9026","Handler":null,"Name":"","Endpoint":""} Jul 07 19:09:38.211336 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.211288 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 19:09:38.211492 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.211346 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 19:09:38.467009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.466896 2568 generic.go:356] "Generic (PLEG): container finished" podID="f7f157e784775de6711a5b9059e36d88" containerID="cb2f6b2d6309e38e557d9146efe5ebd89e908a5c09a448d117cd113cc14e4452" exitCode=0 Jul 07 19:09:38.467009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.466957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" event={"ID":"f7f157e784775de6711a5b9059e36d88","Type":"ContainerDied","Data":"cb2f6b2d6309e38e557d9146efe5ebd89e908a5c09a448d117cd113cc14e4452"} Jul 07 19:09:38.469672 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.469644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" event={"ID":"8c996d6e-041b-4ace-9a6e-898b26328689","Type":"ContainerStarted","Data":"acdaf293764841a6bfc24937c1d1b8b8bb4262de63155449af3756c641378ebe"} Jul 07 19:09:38.476264 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"4836a82e3d96617221497bac649e2e0783dda0d380f4d4ca062628662c66a792"} Jul 07 19:09:38.476444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476273 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"49467a6e7368def345dc4f2a36605075aee378042f1598a4c71216b3945aa958"} Jul 07 19:09:38.476444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"4b034f704d2f3e3ca27e9ea3c339902d9fc27f2b61fbef71d444591c6b90c1c7"} Jul 07 19:09:38.476444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"abceafb298483b94d8ef972be9deb520670cc1697542bdfde699daf90955a9ac"} Jul 07 19:09:38.476444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476405 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"ffb2e741e4c07fecc052ce26f56ca10031b100fbcca8597ec5bbb2d0cc80c16e"} Jul 07 19:09:38.476639 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.476447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"9dfb08f602ba3d9652d2359589ef600bdb9860c52799f1fb5a44d885ee9e4faa"} Jul 07 19:09:38.478047 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:38.477992 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xb9v" event={"ID":"18c0791b-25b2-4c9f-afa4-b49206b43b7a","Type":"ContainerStarted","Data":"31f1581c8d7353ae3a08d1bc325924ff0564921ae849d9c68d95783bbb9ddf88"} Jul 07 19:09:39.345911 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.345814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:39.345911 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.345814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:39.346193 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.345814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:39.346193 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:39.345969 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:39.346193 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:39.346022 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:39.346193 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:39.346076 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:39.482141 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.482102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" event={"ID":"8c996d6e-041b-4ace-9a6e-898b26328689","Type":"ContainerStarted","Data":"451927fd42247ed81766b12313a561716cade9f6730e02fa3617bb857b673f55"} Jul 07 19:09:39.483842 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.483814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" event={"ID":"f7f157e784775de6711a5b9059e36d88","Type":"ContainerStarted","Data":"b400c7ad4d311f84c3e36c8ecfb30280ff203e3ad3d2eb73fd400d47ab4ccc9d"} Jul 07 19:09:39.504636 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.504560 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7w2dp" podStartSLOduration=4.066358913 podStartE2EDuration="24.50453955s" totalImagesPullingTime="20.438180637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:39.503351903 +0000 UTC m=+24.854000140" watchObservedRunningTime="2026-07-07 19:09:39.50453955 +0000 UTC m=+24.855187786" Jul 07 19:09:39.505424 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.505379 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6xb9v" podStartSLOduration=4.562551895 podStartE2EDuration="24.505364673s" totalImagesPullingTime="19.942812778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:38.526048585 +0000 UTC m=+23.876696859" watchObservedRunningTime="2026-07-07 19:09:39.505364673 +0000 UTC m=+24.856012911" Jul 07 19:09:39.520676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:39.520620 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.ec2.internal" podStartSLOduration=23.520602658 podStartE2EDuration="23.520602658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:39.520085299 +0000 UTC m=+24.870733563" watchObservedRunningTime="2026-07-07 19:09:39.520602658 +0000 UTC m=+24.871250895" Jul 07 19:09:40.490994 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:40.490952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"e3c16b4fc3f6b2a050277363df8f34a8007a561558b352ad20117484c6d9cbd8"} Jul 07 19:09:41.346053 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:41.346017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:41.346252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:41.346019 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:41.346252 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:41.346149 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:41.346252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:41.346017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:41.346400 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:41.346246 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:41.346400 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:41.346302 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:43.349373 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.349171 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:43.350063 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.349171 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:43.350063 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:43.349442 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:43.350063 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.349201 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:43.350063 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:43.349518 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:43.350063 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:43.349597 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:43.500876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.500839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" event={"ID":"9b06a6ce-f447-4e16-974f-48c31d448a5a","Type":"ContainerStarted","Data":"cce582a54f559e5bb6f0301c4d7cacdaa06aa999c4fd718f48e47bb49db5ea35"} Jul 07 19:09:43.501249 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.501223 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:43.501366 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.501300 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:43.502770 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.502741 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="17340dc781427f188e6c7d1f79657502b31f0d9d08c6068e7a169fe9552995e0" exitCode=0 Jul 07 19:09:43.502895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.502772 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"17340dc781427f188e6c7d1f79657502b31f0d9d08c6068e7a169fe9552995e0"} Jul 07 19:09:43.517356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.517308 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:43.517505 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.517431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:43.542463 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:43.542396 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" podStartSLOduration=8.612108793 podStartE2EDuration="28.542380641s" totalImagesPullingTime="19.930271848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:43.541802042 +0000 UTC m=+28.892450286" watchObservedRunningTime="2026-07-07 19:09:43.542380641 +0000 UTC m=+28.893028913" Jul 07 19:09:44.504983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:44.504953 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 19:09:45.137329 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.137277 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g4dct"] Jul 07 19:09:45.137498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.137441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:45.137569 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:45.137536 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:45.139981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.139955 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j796b"] Jul 07 19:09:45.140164 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.140050 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:45.140164 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:45.140122 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:45.142958 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.142927 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lk246"] Jul 07 19:09:45.143105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.143061 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:45.143177 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:45.143160 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:45.509191 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.509155 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="b59e5a2a1282fd4ccfbe8265b38e8ee573015d0fae35588ec93e721c00e6977c" exitCode=0 Jul 07 19:09:45.509627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.509230 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"b59e5a2a1282fd4ccfbe8265b38e8ee573015d0fae35588ec93e721c00e6977c"} Jul 07 19:09:45.509627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.509362 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 19:09:45.603683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:45.603641 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:09:46.345967 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.345928 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:46.345967 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.345967 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:46.346228 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.345972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:46.346228 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:46.346045 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:46.346228 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:46.346189 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:46.346355 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:46.346236 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:46.814849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.814642 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:46.815304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.815026 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 19:09:46.815446 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.815426 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h7g62" Jul 07 19:09:46.945106 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:46.945015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.099893 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:47.099852 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.515809 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:47.515688 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="4db0a72d2bff643cf5eaaf3e52314f8e1acdc9c6f4e611181f45e8b30a433bfb" exitCode=0 Jul 07 19:09:47.515809 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:47.515740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"4db0a72d2bff643cf5eaaf3e52314f8e1acdc9c6f4e611181f45e8b30a433bfb"} Jul 07 19:09:47.764421 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:47.764324 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:47.868485 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:47.868449 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:48.346650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.346560 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:48.346821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.346673 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:48.346821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.346684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:48.346821 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.346681 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4dct" podUID="134d60c9-28cc-4320-ac6b-740976d6382d" Jul 07 19:09:48.346821 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.346780 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j796b" podUID="09c946ac-12aa-4b35-9bf6-2f420f357efd" Jul 07 19:09:48.347052 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.346875 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk246" podUID="8120d1f0-2e6a-40e7-8017-7923fce34250" Jul 07 19:09:48.887260 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.887221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:48.887851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.887267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:48.887851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.887378 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:48.887851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.887457 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret podName:134d60c9-28cc-4320-ac6b-740976d6382d nodeName:}" failed. No retries permitted until 2026-07-07 19:10:20.887436836 +0000 UTC m=+66.238085063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret") pod "global-pull-secret-syncer-g4dct" (UID: "134d60c9-28cc-4320-ac6b-740976d6382d") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:09:48.887851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.887385 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:48.887851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.887536 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:20.887519052 +0000 UTC m=+66.238167276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:09:48.988425 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:48.988381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:48.988600 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.988554 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:09:48.988600 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.988576 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:09:48.988600 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.988586 2568 projected.go:196] Error preparing data for projected volume kube-api-access-qgwnq for pod openshift-network-diagnostics/network-check-target-j796b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:48.988694 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:48.988638 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq podName:09c946ac-12aa-4b35-9bf6-2f420f357efd nodeName:}" failed. No retries permitted until 2026-07-07 19:10:20.988624025 +0000 UTC m=+66.339272245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qgwnq" (UniqueName: "kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq") pod "network-check-target-j796b" (UID: "09c946ac-12aa-4b35-9bf6-2f420f357efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:09:49.570842 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.570786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:49.649182 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.649144 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:49.913947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.913907 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-116.ec2.internal" event="NodeReady" Jul 07 19:09:49.914369 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.914046 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 19:09:49.976389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.976357 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-twdjg"] Jul 07 19:09:49.992947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.992813 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wln8w"] Jul 07 19:09:49.993157 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.993123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:49.996210 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.996187 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 19:09:49.996381 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.996282 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g92mm\"" Jul 07 19:09:49.996447 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:49.996414 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 19:09:50.010976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.010941 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twdjg"] Jul 07 19:09:50.011128 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.011000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wln8w"] Jul 07 19:09:50.011128 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.011032 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.013896 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.013870 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nhxfv\"" Jul 07 19:09:50.014090 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.013977 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 19:09:50.014368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.014342 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 19:09:50.014368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.014350 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 19:09:50.095988 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.095887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvx8\" (UniqueName: \"kubernetes.io/projected/4b80486a-4598-4c45-a666-6be9da1b63d9-kube-api-access-gxvx8\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.095988 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.095955 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80486a-4598-4c45-a666-6be9da1b63d9-config-volume\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.096209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.096038 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b80486a-4598-4c45-a666-6be9da1b63d9-tmp-dir\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.096209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.096130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-cert\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.096209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.096199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.096364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.096224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ztf\" (UniqueName: \"kubernetes.io/projected/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-kube-api-access-p9ztf\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.196825 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.196786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80486a-4598-4c45-a666-6be9da1b63d9-config-volume\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.196825 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.196823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b80486a-4598-4c45-a666-6be9da1b63d9-tmp-dir\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.197112 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.196954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-cert\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.197112 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.197039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.197112 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.197066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ztf\" (UniqueName: \"kubernetes.io/projected/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-kube-api-access-p9ztf\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.197112 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.197112 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvx8\" (UniqueName: \"kubernetes.io/projected/4b80486a-4598-4c45-a666-6be9da1b63d9-kube-api-access-gxvx8\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.197308 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:50.197192 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:09:50.197308 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:50.197261 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:50.6972414 +0000 UTC m=+36.047889635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:09:50.197308 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.197276 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b80486a-4598-4c45-a666-6be9da1b63d9-tmp-dir\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.197542 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.197520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80486a-4598-4c45-a666-6be9da1b63d9-config-volume\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.201636 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.201600 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-cert\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.207674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.207646 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvx8\" (UniqueName: \"kubernetes.io/projected/4b80486a-4598-4c45-a666-6be9da1b63d9-kube-api-access-gxvx8\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.207835 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.207721 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ztf\" (UniqueName: \"kubernetes.io/projected/48d69b72-f6ff-4c8b-b14d-fe19db1eb685-kube-api-access-p9ztf\") pod \"ingress-canary-wln8w\" (UID: \"48d69b72-f6ff-4c8b-b14d-fe19db1eb685\") " pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.242853 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.242816 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:50.322628 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.322592 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wln8w" Jul 07 19:09:50.324506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.324471 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:50.345800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.345770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:09:50.345908 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.345803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:09:50.345908 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.345803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:09:50.348975 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.348886 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jkm4d\"" Jul 07 19:09:50.349118 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.349033 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:09:50.349238 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.349220 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:09:50.349297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.349223 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:09:50.349297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.349276 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:09:50.349297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.349284 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jqhn7\"" Jul 07 19:09:50.544115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.543842 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wln8w"] Jul 07 19:09:50.549170 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:09:50.549138 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d69b72_f6ff_4c8b_b14d_fe19db1eb685.slice/crio-a394796199f8ad6029b3f1fd684e1d2fc62ced3a20377e5f2845883739b2ad51 WatchSource:0}: Error finding container a394796199f8ad6029b3f1fd684e1d2fc62ced3a20377e5f2845883739b2ad51: Status 404 returned error can't find the container with id a394796199f8ad6029b3f1fd684e1d2fc62ced3a20377e5f2845883739b2ad51 Jul 07 19:09:50.700712 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:50.700671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:50.700894 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:50.700831 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:09:50.700956 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:50.700896 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:51.700881182 +0000 UTC m=+37.051529397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:09:51.526637 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:51.526602 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wln8w" event={"ID":"48d69b72-f6ff-4c8b-b14d-fe19db1eb685","Type":"ContainerStarted","Data":"a394796199f8ad6029b3f1fd684e1d2fc62ced3a20377e5f2845883739b2ad51"} Jul 07 19:09:51.707109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:51.707073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:51.707279 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:51.707242 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:09:51.707320 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:51.707310 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:53.707292691 +0000 UTC m=+39.057940906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:09:53.722766 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:53.722719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:53.723271 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:53.722874 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:09:53.723271 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:53.722975 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:57.722952937 +0000 UTC m=+43.073601170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:09:54.926399 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:54.926362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:55.024629 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.024594 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:55.488422 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.488390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:55.539696 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.539664 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="f0abd28f1e36a3a1be52fe28e37e20855b158177ff97ede3e81f2de77bf5a280" exitCode=0 Jul 07 19:09:55.539868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.539743 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"f0abd28f1e36a3a1be52fe28e37e20855b158177ff97ede3e81f2de77bf5a280"} Jul 07 19:09:55.541188 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.541157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wln8w" event={"ID":"48d69b72-f6ff-4c8b-b14d-fe19db1eb685","Type":"ContainerStarted","Data":"6e6dcadf015808f9e2125dd015c5cbd000e026bd1d6b6495e9d2bd308c6cf74a"} Jul 07 19:09:55.575648 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.575616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:55.973354 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:55.973310 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:56.044417 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:56.044383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:56.546684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:56.546652 2568 generic.go:356] "Generic (PLEG): container finished" podID="dea3f7e8-5e7b-400f-9eef-551398fe4ad4" containerID="b9bbc9442207e467889a4db40b3fc1fa1a46b50096ab597f5a05042b650e1a4b" exitCode=0 Jul 07 19:09:56.546858 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:56.546726 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerDied","Data":"b9bbc9442207e467889a4db40b3fc1fa1a46b50096ab597f5a05042b650e1a4b"} Jul 07 19:09:56.576159 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:56.576094 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wln8w" podStartSLOduration=3.6945559980000002 podStartE2EDuration="7.576068937s" totalImagesPullingTime="3.881512939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:55.58854015 +0000 UTC m=+40.939188408" watchObservedRunningTime="2026-07-07 19:09:56.576068937 +0000 UTC m=+41.926717177" Jul 07 19:09:57.552119 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:57.551934 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dnk94" event={"ID":"dea3f7e8-5e7b-400f-9eef-551398fe4ad4","Type":"ContainerStarted","Data":"a2ff002f394227bb4828407a6cbd394513c16047a7251864a9f08ca4c65b39e3"} Jul 07 19:09:57.715830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:57.715792 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:09:57.750783 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:57.750750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:09:57.750879 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:57.750859 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:09:57.750936 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:09:57.750911 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:05.750897262 +0000 UTC m=+51.101545478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:09:57.814765 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:09:57.814661 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:01.776121 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:01.776082 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:05.810030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:05.809983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:05.810430 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:05.810139 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:10:05.810430 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:05.810206 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:21.810191398 +0000 UTC m=+67.160839614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:10:09.022742 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.022689 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dnk94" podStartSLOduration=29.305186466 podStartE2EDuration="54.022671795s" totalImagesPullingTime="24.717485329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:57.583562622 +0000 UTC m=+42.934210881" watchObservedRunningTime="2026-07-07 19:10:09.022671795 +0000 UTC m=+54.373320086" Jul 07 19:10:09.023523 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.023502 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-b2pvv"] Jul 07 19:10:09.025530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.025506 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.027965 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.027942 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 19:10:09.028092 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.027970 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 19:10:09.028218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.028206 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 19:10:09.028271 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.028240 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 19:10:09.028804 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.028783 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9klpf\"" Jul 07 19:10:09.032661 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.032839 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032687 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-serving-cert\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.032839 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-tmp\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.032839 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr8g\" (UniqueName: \"kubernetes.io/projected/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-kube-api-access-nhr8g\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.032839 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.033091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.032856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-snapshots\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.033678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.033181 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 19:10:09.037897 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.037868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-b2pvv"] Jul 07 19:10:09.108277 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.108235 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z"] Jul 07 19:10:09.110294 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.110267 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" Jul 07 19:10:09.112769 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.112745 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cq79c\"" Jul 07 19:10:09.127744 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.127709 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z"] Jul 07 19:10:09.133294 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbss\" (UniqueName: \"kubernetes.io/projected/da6d228c-9486-48c6-803b-1ae980b3dd23-kube-api-access-rrbss\") pod \"network-check-source-989f8b8dc-wqp9z\" (UID: \"da6d228c-9486-48c6-803b-1ae980b3dd23\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" Jul 07 19:10:09.133294 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-tmp\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr8g\" (UniqueName: \"kubernetes.io/projected/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-kube-api-access-nhr8g\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133433 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133495 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-snapshots\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-serving-cert\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.133858 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.133837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-tmp\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.134244 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.134221 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.134339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.134290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-snapshots\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.134689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.134671 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.138131 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.138105 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-serving-cert\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.145202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.145164 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr8g\" (UniqueName: \"kubernetes.io/projected/a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758-kube-api-access-nhr8g\") pod \"insights-operator-5c847dcfc4-b2pvv\" (UID: \"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758\") " pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.220861 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.220817 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6"] Jul 07 19:10:09.228729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.228697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:10:09.228867 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.228823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.231283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.231255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 19:10:09.231517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.231499 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lhxzx\"" Jul 07 19:10:09.231609 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.231555 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 19:10:09.232068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.232046 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 19:10:09.232176 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.232108 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:10:09.233503 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.233486 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm"] Jul 07 19:10:09.233636 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.233619 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.234354 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234354 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234354 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-config\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.234533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpkt\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234533 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbss\" (UniqueName: \"kubernetes.io/projected/da6d228c-9486-48c6-803b-1ae980b3dd23-kube-api-access-rrbss\") pod \"network-check-source-989f8b8dc-wqp9z\" (UID: \"da6d228c-9486-48c6-803b-1ae980b3dd23\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" Jul 07 19:10:09.234777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4kl\" (UniqueName: \"kubernetes.io/projected/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-kube-api-access-8c4kl\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.234777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.234777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.234599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.240416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.240382 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6"] Jul 07 19:10:09.240631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.240613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm"] Jul 07 19:10:09.241012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.240995 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" Jul 07 19:10:09.241249 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.241224 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 19:10:09.241558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.240123 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 19:10:09.242155 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.242118 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xhv4z\"" Jul 07 19:10:09.242507 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.242484 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 19:10:09.246185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.246160 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:10:09.246338 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.246166 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 19:10:09.246815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.246792 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xq7fr\"" Jul 07 19:10:09.248156 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.248138 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 19:10:09.255129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.255076 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:10:09.257038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.257014 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbss\" (UniqueName: \"kubernetes.io/projected/da6d228c-9486-48c6-803b-1ae980b3dd23-kube-api-access-rrbss\") pod \"network-check-source-989f8b8dc-wqp9z\" (UID: \"da6d228c-9486-48c6-803b-1ae980b3dd23\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" Jul 07 19:10:09.335379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.335379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflfq\" (UniqueName: \"kubernetes.io/projected/b75f30fe-99ef-46e4-90c6-6ba38e4ec27c-kube-api-access-wflfq\") pod \"volume-data-source-validator-5d456fdbcd-nfdcm\" (UID: \"b75f30fe-99ef-46e4-90c6-6ba38e4ec27c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-config\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.335445 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.335460 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6749c54d-cm8gn: secret "image-registry-tls" not found Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpkt\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.335516 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls podName:61f376af-e5c7-48df-888c-1a82f9581889 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:09.835497247 +0000 UTC m=+55.186145468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls") pod "image-registry-5c6749c54d-cm8gn" (UID: "61f376af-e5c7-48df-888c-1a82f9581889") : secret "image-registry-tls" not found Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.335645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4kl\" (UniqueName: \"kubernetes.io/projected/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-kube-api-access-8c4kl\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.336151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.336151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.336151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.335870 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.336151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.336105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" Jul 07 19:10:09.336578 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.336557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.336676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.336631 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-config\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.337379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.337359 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.338144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.338104 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.338283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.338264 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.338420 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.338402 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.348382 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.348346 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4kl\" (UniqueName: \"kubernetes.io/projected/0a587bbe-ddfc-4ebd-970c-f3b2795ae36c-kube-api-access-8c4kl\") pod \"service-ca-operator-5d46d9ffd4-x44b6\" (UID: \"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.349137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.349080 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.349251 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.349169 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpkt\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.364289 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.364244 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:09.420027 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.419996 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" Jul 07 19:10:09.436740 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.436701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflfq\" (UniqueName: \"kubernetes.io/projected/b75f30fe-99ef-46e4-90c6-6ba38e4ec27c-kube-api-access-wflfq\") pod \"volume-data-source-validator-5d456fdbcd-nfdcm\" (UID: \"b75f30fe-99ef-46e4-90c6-6ba38e4ec27c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" Jul 07 19:10:09.448369 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.448339 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflfq\" (UniqueName: \"kubernetes.io/projected/b75f30fe-99ef-46e4-90c6-6ba38e4ec27c-kube-api-access-wflfq\") pod \"volume-data-source-validator-5d456fdbcd-nfdcm\" (UID: \"b75f30fe-99ef-46e4-90c6-6ba38e4ec27c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" Jul 07 19:10:09.468710 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.468673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:09.511769 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.511737 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-b2pvv"] Jul 07 19:10:09.551374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.551070 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" Jul 07 19:10:09.563608 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.563578 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" Jul 07 19:10:09.581415 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.581304 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" event={"ID":"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758","Type":"ContainerStarted","Data":"3bfb352e4a6bfdfadcc8a545ff5c7699d7f34bfe014b767544c7cdca2cad241e"} Jul 07 19:10:09.586259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.586124 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z"] Jul 07 19:10:09.589566 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:09.589520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6d228c_9486_48c6_803b_1ae980b3dd23.slice/crio-66faa589e3e0f70ddf71bd1549e7e16f324e6f57703945a3e4c3c7b5badf85e1 WatchSource:0}: Error finding container 66faa589e3e0f70ddf71bd1549e7e16f324e6f57703945a3e4c3c7b5badf85e1: Status 404 returned error can't find the container with id 66faa589e3e0f70ddf71bd1549e7e16f324e6f57703945a3e4c3c7b5badf85e1 Jul 07 19:10:09.696491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.696457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6"] Jul 07 19:10:09.699790 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:09.699763 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a587bbe_ddfc_4ebd_970c_f3b2795ae36c.slice/crio-ef43d1d5c29a1fb55a5b19639a6b59019536b509329a73b41d9716ec46127857 WatchSource:0}: Error finding container ef43d1d5c29a1fb55a5b19639a6b59019536b509329a73b41d9716ec46127857: Status 404 returned error can't find the container with id ef43d1d5c29a1fb55a5b19639a6b59019536b509329a73b41d9716ec46127857 Jul 07 19:10:09.713560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.713527 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm"] Jul 07 19:10:09.716408 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:09.716373 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75f30fe_99ef_46e4_90c6_6ba38e4ec27c.slice/crio-a18676367b0fb71f9705741e17b7b0bf18a056973d20fce45bc4bb9b2317048f WatchSource:0}: Error finding container a18676367b0fb71f9705741e17b7b0bf18a056973d20fce45bc4bb9b2317048f: Status 404 returned error can't find the container with id a18676367b0fb71f9705741e17b7b0bf18a056973d20fce45bc4bb9b2317048f Jul 07 19:10:09.839119 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.839025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:09.839266 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.839158 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:10:09.839266 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.839175 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6749c54d-cm8gn: secret "image-registry-tls" not found Jul 07 19:10:09.839266 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:09.839232 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls podName:61f376af-e5c7-48df-888c-1a82f9581889 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:10.839218284 +0000 UTC m=+56.189866499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls") pod "image-registry-5c6749c54d-cm8gn" (UID: "61f376af-e5c7-48df-888c-1a82f9581889") : secret "image-registry-tls" not found Jul 07 19:10:09.892668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.892615 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:09.987195 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:09.987155 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:10.090870 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:10.090767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:10.585368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:10.585320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" event={"ID":"da6d228c-9486-48c6-803b-1ae980b3dd23","Type":"ContainerStarted","Data":"66faa589e3e0f70ddf71bd1549e7e16f324e6f57703945a3e4c3c7b5badf85e1"} Jul 07 19:10:10.587067 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:10.587018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" event={"ID":"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c","Type":"ContainerStarted","Data":"ef43d1d5c29a1fb55a5b19639a6b59019536b509329a73b41d9716ec46127857"} Jul 07 19:10:10.589577 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:10.589514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" event={"ID":"b75f30fe-99ef-46e4-90c6-6ba38e4ec27c","Type":"ContainerStarted","Data":"a18676367b0fb71f9705741e17b7b0bf18a056973d20fce45bc4bb9b2317048f"} Jul 07 19:10:10.845818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:10.845702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:10.846017 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:10.845827 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:10:10.846017 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:10.845846 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6749c54d-cm8gn: secret "image-registry-tls" not found Jul 07 19:10:10.846017 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:10.845941 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls podName:61f376af-e5c7-48df-888c-1a82f9581889 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:12.845900885 +0000 UTC m=+58.196549111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls") pod "image-registry-5c6749c54d-cm8gn" (UID: "61f376af-e5c7-48df-888c-1a82f9581889") : secret "image-registry-tls" not found Jul 07 19:10:12.598551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.598491 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" event={"ID":"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758","Type":"ContainerStarted","Data":"cfc92b3a54a5f496706400de1562d6f8ff8bf7855885d28e986f3da1dcb33392"} Jul 07 19:10:12.600235 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.600195 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" event={"ID":"b75f30fe-99ef-46e4-90c6-6ba38e4ec27c","Type":"ContainerStarted","Data":"50edc4845179da89f9d4b5f2dee01a28b7c91d19add73aa85e7cde9d9241a2e3"} Jul 07 19:10:12.634745 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.634683 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-nfdcm" podStartSLOduration=1.491328053 podStartE2EDuration="3.634662546s" totalImagesPullingTime="2.143334493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:12.633693297 +0000 UTC m=+57.984341589" watchObservedRunningTime="2026-07-07 19:10:12.634662546 +0000 UTC m=+57.985310788" Jul 07 19:10:12.634988 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.634953 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" podStartSLOduration=2.266756638 podStartE2EDuration="4.634942858s" totalImagesPullingTime="2.36818622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:12.61832881 +0000 UTC m=+57.968977049" watchObservedRunningTime="2026-07-07 19:10:12.634942858 +0000 UTC m=+57.985591096" Jul 07 19:10:12.823817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.823779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:12.862218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.862189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:12.862424 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:12.862337 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:10:12.862424 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:12.862352 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6749c54d-cm8gn: secret "image-registry-tls" not found Jul 07 19:10:12.862589 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:12.862433 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls podName:61f376af-e5c7-48df-888c-1a82f9581889 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:16.862414276 +0000 UTC m=+62.213062490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls") pod "image-registry-5c6749c54d-cm8gn" (UID: "61f376af-e5c7-48df-888c-1a82f9581889") : secret "image-registry-tls" not found Jul 07 19:10:12.924434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:12.924394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:13.168500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.168410 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-njvmk"] Jul 07 19:10:13.173908 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.173878 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.176256 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.176230 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 19:10:13.176416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.176344 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 19:10:13.176416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.176382 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f8xvk\"" Jul 07 19:10:13.185468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.185419 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njvmk"] Jul 07 19:10:13.207867 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.207822 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:13.265145 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.265108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdn8\" (UniqueName: \"kubernetes.io/projected/da03f687-4e8f-4302-bbc4-b203e1862856-kube-api-access-msdn8\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.265344 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.265165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da03f687-4e8f-4302-bbc4-b203e1862856-data-volume\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.265344 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.265297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da03f687-4e8f-4302-bbc4-b203e1862856-crio-socket\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.265344 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.265330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.265498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.265411 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.281975 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.281913 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:13.366413 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da03f687-4e8f-4302-bbc4-b203e1862856-crio-socket\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366433 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdn8\" (UniqueName: \"kubernetes.io/projected/da03f687-4e8f-4302-bbc4-b203e1862856-kube-api-access-msdn8\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da03f687-4e8f-4302-bbc4-b203e1862856-data-volume\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.366536 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da03f687-4e8f-4302-bbc4-b203e1862856-crio-socket\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.366851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.366637 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:10:13.366851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.366681 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:10:13.366851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.366719 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:13.866697534 +0000 UTC m=+59.217345757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : configmap "kube-rbac-proxy" not found Jul 07 19:10:13.366851 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.366763 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:13.866752727 +0000 UTC m=+59.217400946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : secret "insights-runtime-extractor-tls" not found Jul 07 19:10:13.367087 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.367032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da03f687-4e8f-4302-bbc4-b203e1862856-data-volume\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.378273 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.378240 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdn8\" (UniqueName: \"kubernetes.io/projected/da03f687-4e8f-4302-bbc4-b203e1862856-kube-api-access-msdn8\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.871812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.871767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.872265 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:13.871823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:13.872265 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.871910 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:10:13.872265 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.871958 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:10:13.872265 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.871995 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:14.871973959 +0000 UTC m=+60.222622182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : configmap "kube-rbac-proxy" not found Jul 07 19:10:13.872265 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:13.872023 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:14.872009901 +0000 UTC m=+60.222658117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : secret "insights-runtime-extractor-tls" not found Jul 07 19:10:14.347998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.347880 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:14.428777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.428739 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:14.606574 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.606479 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" event={"ID":"da6d228c-9486-48c6-803b-1ae980b3dd23","Type":"ContainerStarted","Data":"8c7b6a5f04d0e2d128b5e3317588cb0e2c1f9684bae5158704825ba2aa576dc1"} Jul 07 19:10:14.608033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.608003 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" event={"ID":"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c","Type":"ContainerStarted","Data":"2d9edbdaf3509eadae04415f1ef3e93c7e151cd4fc54b3615b541991f7cc1f12"} Jul 07 19:10:14.640181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.640125 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" podStartSLOduration=1.306227814 podStartE2EDuration="5.640106771s" totalImagesPullingTime="4.333878957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:14.639542439 +0000 UTC m=+59.990190675" watchObservedRunningTime="2026-07-07 19:10:14.640106771 +0000 UTC m=+59.990755009" Jul 07 19:10:14.644935 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.644853 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wqp9z" podStartSLOduration=1.196598349 podStartE2EDuration="5.644833423s" totalImagesPullingTime="4.448235074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:14.623538153 +0000 UTC m=+59.974186403" watchObservedRunningTime="2026-07-07 19:10:14.644833423 +0000 UTC m=+59.995481659" Jul 07 19:10:14.880200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.880159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:14.880200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:14.880204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:14.880679 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:14.880274 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:10:14.880679 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:14.880328 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:16.880314478 +0000 UTC m=+62.230962696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : configmap "kube-rbac-proxy" not found Jul 07 19:10:14.880679 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:14.880344 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:10:14.880679 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:14.880421 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:16.880400539 +0000 UTC m=+62.231048759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : secret "insights-runtime-extractor-tls" not found Jul 07 19:10:15.156993 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:15.156899 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2p956_6e39206e-6b9b-4a92-a0ed-705902137faf/dns-node-resolver/0.log" Jul 07 19:10:15.374871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:15.374828 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:15.453671 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:15.453570 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:15.871230 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:15.871197 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:15.969643 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:15.969614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:16.086576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.086533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:16.186103 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.186019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:16.357949 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.357905 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xhfcg_2dfb8567-19e4-48fa-98a5-68e65b8a3378/node-ca/0.log" Jul 07 19:10:16.526618 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.526537 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4z7b4" Jul 07 19:10:16.689439 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.689404 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:16.786986 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.786880 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:16.898178 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.898139 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.898190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:16.898214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898289 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898293 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898318 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6749c54d-cm8gn: secret "image-registry-tls" not found Jul 07 19:10:16.898358 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898340 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:20.89832585 +0000 UTC m=+66.248974070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : configmap "kube-rbac-proxy" not found Jul 07 19:10:16.898550 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898373 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls podName:61f376af-e5c7-48df-888c-1a82f9581889 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:24.898360308 +0000 UTC m=+70.249008524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls") pod "image-registry-5c6749c54d-cm8gn" (UID: "61f376af-e5c7-48df-888c-1a82f9581889") : secret "image-registry-tls" not found Jul 07 19:10:16.898550 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898300 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:10:16.898550 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:16.898448 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:20.898435848 +0000 UTC m=+66.249084063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : secret "insights-runtime-extractor-tls" not found Jul 07 19:10:17.161308 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.161283 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wln8w_48d69b72-f6ff-4c8b-b14d-fe19db1eb685/serve-healthcheck-canary/0.log" Jul 07 19:10:17.670806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.670770 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gvsxb"] Jul 07 19:10:17.673784 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.673761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.675993 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.675971 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 19:10:17.676149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.676078 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 19:10:17.676149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.676084 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 19:10:17.676149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.676113 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 19:10:17.676149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.676078 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-62r8b\"" Jul 07 19:10:17.682325 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.682296 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gvsxb"] Jul 07 19:10:17.703316 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.703279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-key\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.703490 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.703401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-cabundle\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.703490 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.703446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tswr\" (UniqueName: \"kubernetes.io/projected/b2e8e1fe-f98b-47ed-8d50-e461db74550a-kube-api-access-8tswr\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.706871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.706839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:17.792397 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.792365 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:17.804383 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.804348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tswr\" (UniqueName: \"kubernetes.io/projected/b2e8e1fe-f98b-47ed-8d50-e461db74550a-kube-api-access-8tswr\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.804529 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.804416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-key\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.804573 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.804533 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-cabundle\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.805198 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.805179 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-cabundle\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.806963 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.806942 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2e8e1fe-f98b-47ed-8d50-e461db74550a-signing-key\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.815766 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.815737 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tswr\" (UniqueName: \"kubernetes.io/projected/b2e8e1fe-f98b-47ed-8d50-e461db74550a-kube-api-access-8tswr\") pod \"service-ca-758f599c5-gvsxb\" (UID: \"b2e8e1fe-f98b-47ed-8d50-e461db74550a\") " pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:17.984200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:17.984106 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-gvsxb" Jul 07 19:10:18.114381 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.114349 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gvsxb"] Jul 07 19:10:18.117445 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:18.117404 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e8e1fe_f98b_47ed_8d50_e461db74550a.slice/crio-5ea2b6da78e56d29d95699931e807f67c46417526a9e0b87a5eee7f2e95a2028 WatchSource:0}: Error finding container 5ea2b6da78e56d29d95699931e807f67c46417526a9e0b87a5eee7f2e95a2028: Status 404 returned error can't find the container with id 5ea2b6da78e56d29d95699931e807f67c46417526a9e0b87a5eee7f2e95a2028 Jul 07 19:10:18.554039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.554010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:18.621322 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.621293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-gvsxb" event={"ID":"b2e8e1fe-f98b-47ed-8d50-e461db74550a","Type":"ContainerStarted","Data":"96719efb7c34941ff53fc25c557f1b93fdbe4f861f0f0fdb29ba902524de2873"} Jul 07 19:10:18.621503 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.621339 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-gvsxb" event={"ID":"b2e8e1fe-f98b-47ed-8d50-e461db74550a","Type":"ContainerStarted","Data":"5ea2b6da78e56d29d95699931e807f67c46417526a9e0b87a5eee7f2e95a2028"} Jul 07 19:10:18.641283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.641228 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-gvsxb" podStartSLOduration=1.6412093639999998 podStartE2EDuration="1.641209364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:18.639960188 +0000 UTC m=+63.990608446" watchObservedRunningTime="2026-07-07 19:10:18.641209364 +0000 UTC m=+63.991857602" Jul 07 19:10:18.669027 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.668980 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:18.880463 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:18.880430 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:19.137902 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:19.137808 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:19.243889 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:19.243850 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:19.347179 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:19.347140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:19.446430 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:19.446332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:20.021292 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.021256 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:20.812186 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.812153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:20.931655 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.931614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.931663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.931725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.931746 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.931760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.931783 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.931819 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:28.931797903 +0000 UTC m=+74.282446123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : configmap "kube-rbac-proxy" not found Jul 07 19:10:20.931867 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.931845 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls podName:da03f687-4e8f-4302-bbc4-b203e1862856 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:28.931828684 +0000 UTC m=+74.282476898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njvmk" (UID: "da03f687-4e8f-4302-bbc4-b203e1862856") : secret "insights-runtime-extractor-tls" not found Jul 07 19:10:20.934181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.934154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:10:20.934310 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.934189 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:10:20.942396 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.942357 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 19:10:20.942504 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:20.942444 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs podName:8120d1f0-2e6a-40e7-8017-7923fce34250 nodeName:}" failed. No retries permitted until 2026-07-07 19:11:24.942420997 +0000 UTC m=+130.293069213 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs") pod "network-metrics-daemon-lk246" (UID: "8120d1f0-2e6a-40e7-8017-7923fce34250") : secret "metrics-daemon-secret" not found Jul 07 19:10:20.944632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.944605 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/134d60c9-28cc-4320-ac6b-740976d6382d-original-pull-secret\") pod \"global-pull-secret-syncer-g4dct\" (UID: \"134d60c9-28cc-4320-ac6b-740976d6382d\") " pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:10:20.977009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:20.976966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4dct" Jul 07 19:10:21.033674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.033646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:10:21.036965 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.036840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwnq\" (UniqueName: \"kubernetes.io/projected/09c946ac-12aa-4b35-9bf6-2f420f357efd-kube-api-access-qgwnq\") pod \"network-check-target-j796b\" (UID: \"09c946ac-12aa-4b35-9bf6-2f420f357efd\") " pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:10:21.115539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.115511 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g4dct"] Jul 07 19:10:21.117983 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:21.117949 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134d60c9_28cc_4320_ac6b_740976d6382d.slice/crio-42ba4f28f07ed1e38c20c2b5facf706734cef9acc31cf178ee57814db4d0dfb1 WatchSource:0}: Error finding container 42ba4f28f07ed1e38c20c2b5facf706734cef9acc31cf178ee57814db4d0dfb1: Status 404 returned error can't find the container with id 42ba4f28f07ed1e38c20c2b5facf706734cef9acc31cf178ee57814db4d0dfb1 Jul 07 19:10:21.216282 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.216236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:21.284159 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.284122 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jkm4d\"" Jul 07 19:10:21.292185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.292156 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:10:21.413748 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.413696 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:21.428468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.428430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j796b"] Jul 07 19:10:21.432804 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:21.432773 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c946ac_12aa_4b35_9bf6_2f420f357efd.slice/crio-409307871e63ffcd693ce4e452fef173ac2112a8fc2bab03fae0fe8b5a1214cd WatchSource:0}: Error finding container 409307871e63ffcd693ce4e452fef173ac2112a8fc2bab03fae0fe8b5a1214cd: Status 404 returned error can't find the container with id 409307871e63ffcd693ce4e452fef173ac2112a8fc2bab03fae0fe8b5a1214cd Jul 07 19:10:21.611213 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.611178 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:21.633194 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.633150 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j796b" event={"ID":"09c946ac-12aa-4b35-9bf6-2f420f357efd","Type":"ContainerStarted","Data":"bf977ca166b0847faa31b03a6a3d804b1efe7d50763d90463eefcb1f60c8d53a"} Jul 07 19:10:21.633363 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.633215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j796b" event={"ID":"09c946ac-12aa-4b35-9bf6-2f420f357efd","Type":"ContainerStarted","Data":"409307871e63ffcd693ce4e452fef173ac2112a8fc2bab03fae0fe8b5a1214cd"} Jul 07 19:10:21.633363 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.633276 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:10:21.634755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.634715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g4dct" event={"ID":"134d60c9-28cc-4320-ac6b-740976d6382d","Type":"ContainerStarted","Data":"42ba4f28f07ed1e38c20c2b5facf706734cef9acc31cf178ee57814db4d0dfb1"} Jul 07 19:10:21.652933 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.652862 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j796b" podStartSLOduration=66.652839136 podStartE2EDuration="1m6.652839136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:21.651409988 +0000 UTC m=+67.002058248" watchObservedRunningTime="2026-07-07 19:10:21.652839136 +0000 UTC m=+67.003487374" Jul 07 19:10:21.812777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.812677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:21.841697 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.841661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:21.841878 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:21.841821 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:10:21.841978 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:21.841891 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls podName:4b80486a-4598-4c45-a666-6be9da1b63d9 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:53.841870059 +0000 UTC m=+99.192518274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls") pod "dns-default-twdjg" (UID: "4b80486a-4598-4c45-a666-6be9da1b63d9") : secret "dns-default-metrics-tls" not found Jul 07 19:10:21.913610 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:21.913495 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:22.609640 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:22.609601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:22.796484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:22.796442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:22.864020 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:22.863907 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:23.111813 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:23.111768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:23.213009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:23.212903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:23.407061 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:23.406998 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:23.608756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:23.608726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.010602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:24.010499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.075823 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:24.075787 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.612689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:24.612658 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:24.973245 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:24.973152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:24.976747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:24.976713 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"image-registry-5c6749c54d-cm8gn\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:25.014631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.014595 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:25.154895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.154859 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xhv4z\"" Jul 07 19:10:25.162893 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.162863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:25.411427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.411371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:25.461464 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.461430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:10:25.466222 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:25.466188 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f376af_e5c7_48df_888c_1a82f9581889.slice/crio-6e1b298a22e4b9afe3cb52b3f523744827ded0d87bbae807ab7f249b4efa8f3e WatchSource:0}: Error finding container 6e1b298a22e4b9afe3cb52b3f523744827ded0d87bbae807ab7f249b4efa8f3e: Status 404 returned error can't find the container with id 6e1b298a22e4b9afe3cb52b3f523744827ded0d87bbae807ab7f249b4efa8f3e Jul 07 19:10:25.608603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.608506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:25.652418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.652377 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g4dct" event={"ID":"134d60c9-28cc-4320-ac6b-740976d6382d","Type":"ContainerStarted","Data":"a0cf82f2bc35cee7d296a4323cdd037f142ca4e0587e7621b0ee2b4acc2a1a0b"} Jul 07 19:10:25.654626 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.654594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" event={"ID":"61f376af-e5c7-48df-888c-1a82f9581889","Type":"ContainerStarted","Data":"7990431ebc74bb0c241c3e036487640cf79896b43a4a71f05d2d4f25be5786a5"} Jul 07 19:10:25.654786 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.654646 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" event={"ID":"61f376af-e5c7-48df-888c-1a82f9581889","Type":"ContainerStarted","Data":"6e1b298a22e4b9afe3cb52b3f523744827ded0d87bbae807ab7f249b4efa8f3e"} Jul 07 19:10:25.655206 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.655188 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:25.671876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.671812 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g4dct" podStartSLOduration=66.440454546 podStartE2EDuration="1m10.671789695s" totalImagesPullingTime="4.231335149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:25.670676905 +0000 UTC m=+71.021325140" watchObservedRunningTime="2026-07-07 19:10:25.671789695 +0000 UTC m=+71.022437932" Jul 07 19:10:25.691727 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:25.691672 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" podStartSLOduration=16.691650494 podStartE2EDuration="16.691650494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:25.690680995 +0000 UTC m=+71.041329263" watchObservedRunningTime="2026-07-07 19:10:25.691650494 +0000 UTC m=+71.042298761" Jul 07 19:10:26.012013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:26.011900 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:26.219056 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:26.219023 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:26.320619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:26.320529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:26.422109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:26.422080 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:27.416267 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:27.416228 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:27.608745 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:27.608709 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:27.813044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:27.812949 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:28.018728 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:28.018686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:28.221286 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:28.221247 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.009115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.009075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:29.009115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.009114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:29.010014 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.009634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da03f687-4e8f-4302-bbc4-b203e1862856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:29.011624 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.011595 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da03f687-4e8f-4302-bbc4-b203e1862856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njvmk\" (UID: \"da03f687-4e8f-4302-bbc4-b203e1862856\") " pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:29.088328 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.088298 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f8xvk\"" Jul 07 19:10:29.097021 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.096995 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njvmk" Jul 07 19:10:29.212970 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.212911 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.263932 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.263832 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njvmk"] Jul 07 19:10:29.266893 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:29.266854 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda03f687_4e8f_4302_bbc4_b203e1862856.slice/crio-1fd250a73a00752a7b0e129210a6f837be26719206ab8b4302d9d49264e57d42 WatchSource:0}: Error finding container 1fd250a73a00752a7b0e129210a6f837be26719206ab8b4302d9d49264e57d42: Status 404 returned error can't find the container with id 1fd250a73a00752a7b0e129210a6f837be26719206ab8b4302d9d49264e57d42 Jul 07 19:10:29.412512 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.412473 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.621476 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.621439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.669109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.669073 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njvmk" event={"ID":"da03f687-4e8f-4302-bbc4-b203e1862856","Type":"ContainerStarted","Data":"1fd250a73a00752a7b0e129210a6f837be26719206ab8b4302d9d49264e57d42"} Jul 07 19:10:29.779452 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.779421 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:29.855621 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:29.855589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:33.685983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:33.685943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njvmk" event={"ID":"da03f687-4e8f-4302-bbc4-b203e1862856","Type":"ContainerStarted","Data":"744aaa40fdad2f1715dc78084e9d32e30f77b87119f37f0dc993d50a69e4e933"} Jul 07 19:10:33.849799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:33.849762 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:33.951320 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:33.951226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.021873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.021838 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.358660 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.358525 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.524147 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.524111 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.624863 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.624717 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.691573 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.691534 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njvmk" event={"ID":"da03f687-4e8f-4302-bbc4-b203e1862856","Type":"ContainerStarted","Data":"a252e7dcffdf48d4fc8c6832ec7d6d441f627f6e02cd992a5c8aee40ef639b2c"} Jul 07 19:10:34.742320 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.742272 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:34.809971 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:34.809928 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:35.240424 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:35.240391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:35.341222 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:35.341165 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:35.538084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:35.537982 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:35.944211 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:35.944175 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:36.169082 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:36.169028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:36.269600 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:36.269408 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:36.700384 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:36.700351 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njvmk" event={"ID":"da03f687-4e8f-4302-bbc4-b203e1862856","Type":"ContainerStarted","Data":"a59d1517d30312c3b1c3fb744a201cf844615fc4e8e89e9f03cf17d0b40a9eb9"} Jul 07 19:10:36.718684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:36.718622 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-njvmk" podStartSLOduration=16.759891034 podStartE2EDuration="23.718603492s" totalImagesPullingTime="6.958712458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:10:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:36.718148151 +0000 UTC m=+82.068796388" watchObservedRunningTime="2026-07-07 19:10:36.718603492 +0000 UTC m=+82.069251732" Jul 07 19:10:36.771733 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:36.771699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.165135 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.165086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.266301 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.266274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.593509 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.593419 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx"] Jul 07 19:10:41.596351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.596328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.598490 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.598468 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 19:10:41.599149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.599130 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 19:10:41.599252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.599154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9t7pr\"" Jul 07 19:10:41.601741 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.601704 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8"] Jul 07 19:10:41.603781 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.603762 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.605801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.605782 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 19:10:41.605978 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.605942 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 19:10:41.606052 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.606017 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 19:10:41.606178 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.606164 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 19:10:41.606282 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.606267 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nh57j\"" Jul 07 19:10:41.612245 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.612220 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx"] Jul 07 19:10:41.616120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.616098 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8"] Jul 07 19:10:41.618725 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.618701 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:10:41.623436 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.623379 2568 patch_prober.go:28] interesting pod/image-registry-5c6749c54d-cm8gn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 19:10:41.623588 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.623456 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" podUID="61f376af-e5c7-48df-888c-1a82f9581889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:10:41.695810 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.695770 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f85858d46-w926r"] Jul 07 19:10:41.697974 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.697955 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.712571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.712539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c08a6d92-0b57-4e67-9c9a-271e6b51755c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.712762 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.712617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c08a6d92-0b57-4e67-9c9a-271e6b51755c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.712762 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.712678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6pj\" (UniqueName: \"kubernetes.io/projected/11de426b-5f13-4b53-b616-ff5266d38ad9-kube-api-access-gq6pj\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.712762 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.712742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11de426b-5f13-4b53-b616-ff5266d38ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.719860 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.719817 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f85858d46-w926r"] Jul 07 19:10:41.726826 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.726789 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.813620 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11de426b-5f13-4b53-b616-ff5266d38ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.813812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-image-registry-private-configuration\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.813812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp98\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-kube-api-access-rnp98\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.813812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-trusted-ca\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.813812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c08a6d92-0b57-4e67-9c9a-271e6b51755c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.813812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-installation-pull-secrets\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-ca-trust-extracted\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c08a6d92-0b57-4e67-9c9a-271e6b51755c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-tls\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6pj\" (UniqueName: \"kubernetes.io/projected/11de426b-5f13-4b53-b616-ff5266d38ad9-kube-api-access-gq6pj\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-certificates\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.814048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.813990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-bound-sa-token\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.814588 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.814562 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c08a6d92-0b57-4e67-9c9a-271e6b51755c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.816239 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.816207 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c08a6d92-0b57-4e67-9c9a-271e6b51755c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tfx\" (UID: \"c08a6d92-0b57-4e67-9c9a-271e6b51755c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.816358 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.816302 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11de426b-5f13-4b53-b616-ff5266d38ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.829083 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.829040 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6pj\" (UniqueName: \"kubernetes.io/projected/11de426b-5f13-4b53-b616-ff5266d38ad9-kube-api-access-gq6pj\") pod \"managed-serviceaccount-addon-agent-597f84b66-gdkq8\" (UID: \"11de426b-5f13-4b53-b616-ff5266d38ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:41.829645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.829615 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:41.905232 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.905197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" Jul 07 19:10:41.914619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.914594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-certificates\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.914726 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.914633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-bound-sa-token\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.914825 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.914797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-image-registry-private-configuration\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.914882 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.914857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp98\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-kube-api-access-rnp98\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.914977 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.914889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-trusted-ca\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915067 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-installation-pull-secrets\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915139 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-ca-trust-extracted\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915191 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-tls\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915511 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-certificates\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915647 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-ca-trust-extracted\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.915996 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.915969 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-trusted-ca\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.917952 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.917899 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-registry-tls\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.918054 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.917994 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-image-registry-private-configuration\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.918144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.918121 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-installation-pull-secrets\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.923330 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.923299 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-bound-sa-token\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.923427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.923405 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp98\" (UniqueName: \"kubernetes.io/projected/25a5a89f-b332-4fc6-a920-f6ae0ec6aa31-kube-api-access-rnp98\") pod \"image-registry-5f85858d46-w926r\" (UID: \"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31\") " pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:41.928143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:41.928116 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" Jul 07 19:10:42.007345 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.007308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:42.055742 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.055707 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx"] Jul 07 19:10:42.059595 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:42.059563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08a6d92_0b57_4e67_9c9a_271e6b51755c.slice/crio-26ed42a2c5b34c5b4360883a43042a9629b3884945a4f42c5a5d9e0eb8217765 WatchSource:0}: Error finding container 26ed42a2c5b34c5b4360883a43042a9629b3884945a4f42c5a5d9e0eb8217765: Status 404 returned error can't find the container with id 26ed42a2c5b34c5b4360883a43042a9629b3884945a4f42c5a5d9e0eb8217765 Jul 07 19:10:42.077956 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.077892 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8"] Jul 07 19:10:42.081433 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:42.081397 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11de426b_5f13_4b53_b616_ff5266d38ad9.slice/crio-72b07a9f80edbcf6b9364b890603438c4c6cb56974fe98b9c03450d108e1695b WatchSource:0}: Error finding container 72b07a9f80edbcf6b9364b890603438c4c6cb56974fe98b9c03450d108e1695b: Status 404 returned error can't find the container with id 72b07a9f80edbcf6b9364b890603438c4c6cb56974fe98b9c03450d108e1695b Jul 07 19:10:42.156655 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.156591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f85858d46-w926r"] Jul 07 19:10:42.159064 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:42.159042 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a5a89f_b332_4fc6_a920_f6ae0ec6aa31.slice/crio-aa0175cbd6e1c2bbc9f2c0ec7ed3828768377f2316bf00a764bb5d7aa3542611 WatchSource:0}: Error finding container aa0175cbd6e1c2bbc9f2c0ec7ed3828768377f2316bf00a764bb5d7aa3542611: Status 404 returned error can't find the container with id aa0175cbd6e1c2bbc9f2c0ec7ed3828768377f2316bf00a764bb5d7aa3542611 Jul 07 19:10:42.721669 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.721627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f85858d46-w926r" event={"ID":"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31","Type":"ContainerStarted","Data":"a1e34675cb2ff10297f1e7dbf48d389782e53657ecc8c41e080afb8f0fcfb3c9"} Jul 07 19:10:42.722060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.721691 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f85858d46-w926r" event={"ID":"25a5a89f-b332-4fc6-a920-f6ae0ec6aa31","Type":"ContainerStarted","Data":"aa0175cbd6e1c2bbc9f2c0ec7ed3828768377f2316bf00a764bb5d7aa3542611"} Jul 07 19:10:42.722332 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.722274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:10:42.725165 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.725097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" event={"ID":"c08a6d92-0b57-4e67-9c9a-271e6b51755c","Type":"ContainerStarted","Data":"26ed42a2c5b34c5b4360883a43042a9629b3884945a4f42c5a5d9e0eb8217765"} Jul 07 19:10:42.728664 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.728621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" event={"ID":"11de426b-5f13-4b53-b616-ff5266d38ad9","Type":"ContainerStarted","Data":"72b07a9f80edbcf6b9364b890603438c4c6cb56974fe98b9c03450d108e1695b"} Jul 07 19:10:42.747804 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:42.747740 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f85858d46-w926r" podStartSLOduration=1.7477181389999998 podStartE2EDuration="1.747718139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:42.747639568 +0000 UTC m=+88.098287828" watchObservedRunningTime="2026-07-07 19:10:42.747718139 +0000 UTC m=+88.098366373" Jul 07 19:10:44.737541 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.737502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" event={"ID":"c08a6d92-0b57-4e67-9c9a-271e6b51755c","Type":"ContainerStarted","Data":"e2b078aa971c703d54c3b3a4519e6a35c62835d4aeb1262f5444a3d5f175f957"} Jul 07 19:10:44.739116 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.739083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" event={"ID":"11de426b-5f13-4b53-b616-ff5266d38ad9","Type":"ContainerStarted","Data":"9cf4a696291219a3c0c5ba28b13dd62e8a86abd1ee3197b4db4f7c68d90184c8"} Jul 07 19:10:44.757549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.757491 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tfx" podStartSLOduration=1.517172225 podStartE2EDuration="3.757473113s" totalImagesPullingTime="2.240300888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:44.757037571 +0000 UTC m=+90.107685808" watchObservedRunningTime="2026-07-07 19:10:44.757473113 +0000 UTC m=+90.108121328" Jul 07 19:10:44.769880 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.769846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:44.775353 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.775292 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-597f84b66-gdkq8" podStartSLOduration=1.5047246539999999 podStartE2EDuration="3.775268334s" totalImagesPullingTime="2.27054368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:44.774122107 +0000 UTC m=+90.124770344" watchObservedRunningTime="2026-07-07 19:10:44.775268334 +0000 UTC m=+90.125916572" Jul 07 19:10:44.866364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:44.866315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:45.047354 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:45.047261 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:45.312744 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:45.312654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:45.413490 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:45.413444 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:45.493638 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:45.493603 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:45.608016 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:45.607896 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:46.773525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:46.773490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:46.839368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:46.839336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:49.512983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.512943 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:49.557775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.557741 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs"] Jul 07 19:10:49.561142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.561124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.563568 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.563541 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 19:10:49.564259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.564236 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 19:10:49.564389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.564262 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 19:10:49.564389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.564291 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8b9s9\"" Jul 07 19:10:49.564389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.564309 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 19:10:49.564645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.564627 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 19:10:49.572540 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.572514 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs"] Jul 07 19:10:49.598550 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.598506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:49.679791 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.679752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.679998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.679812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.679998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.679871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728513c9-ccf1-4591-8a05-43988359a583-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.680090 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.680003 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxdj\" (UniqueName: \"kubernetes.io/projected/728513c9-ccf1-4591-8a05-43988359a583-kube-api-access-pcxdj\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.780895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.780802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728513c9-ccf1-4591-8a05-43988359a583-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.780895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.780870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxdj\" (UniqueName: \"kubernetes.io/projected/728513c9-ccf1-4591-8a05-43988359a583-kube-api-access-pcxdj\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.781130 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.780952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.781130 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.781012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.781130 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:49.781086 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 19:10:49.781246 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:49.781166 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls podName:728513c9-ccf1-4591-8a05-43988359a583 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:50.281148127 +0000 UTC m=+95.631796342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-4m5qs" (UID: "728513c9-ccf1-4591-8a05-43988359a583") : secret "prometheus-operator-tls" not found Jul 07 19:10:49.781661 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.781635 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/728513c9-ccf1-4591-8a05-43988359a583-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.783519 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.783482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.791235 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.791209 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxdj\" (UniqueName: \"kubernetes.io/projected/728513c9-ccf1-4591-8a05-43988359a583-kube-api-access-pcxdj\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:49.929607 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:49.929567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:50.286636 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.286581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:50.288983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.288952 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/728513c9-ccf1-4591-8a05-43988359a583-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-4m5qs\" (UID: \"728513c9-ccf1-4591-8a05-43988359a583\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:50.470236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.470191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" Jul 07 19:10:50.545085 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.544839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:50.603821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.603788 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs"] Jul 07 19:10:50.607528 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:50.607493 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728513c9_ccf1_4591_8a05_43988359a583.slice/crio-819bc233a4ff2ccd9542b5d815396de5805ab7a8f5cf53dd51134b4be45f7d85 WatchSource:0}: Error finding container 819bc233a4ff2ccd9542b5d815396de5805ab7a8f5cf53dd51134b4be45f7d85: Status 404 returned error can't find the container with id 819bc233a4ff2ccd9542b5d815396de5805ab7a8f5cf53dd51134b4be45f7d85 Jul 07 19:10:50.760872 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.760837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" event={"ID":"728513c9-ccf1-4591-8a05-43988359a583","Type":"ContainerStarted","Data":"819bc233a4ff2ccd9542b5d815396de5805ab7a8f5cf53dd51134b4be45f7d85"} Jul 07 19:10:50.771894 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.771866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:50.854247 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:50.854153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:51.315604 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:51.315563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:51.417809 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:51.417777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:51.624068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:51.624043 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:10:52.250323 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:52.250231 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:52.640448 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:52.640421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j796b" Jul 07 19:10:52.769730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:52.769698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" event={"ID":"728513c9-ccf1-4591-8a05-43988359a583","Type":"ContainerStarted","Data":"e04d47367b84bfaf05facd3c89c41ab3ca9bc61dd819e95c597c0b052664e2d5"} Jul 07 19:10:52.769909 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:52.769751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" event={"ID":"728513c9-ccf1-4591-8a05-43988359a583","Type":"ContainerStarted","Data":"c27976025843545226a602b51ca6e565559edc848095e922ea252da5a65cfe76"} Jul 07 19:10:52.793156 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:52.793105 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-4m5qs" podStartSLOduration=2.486158 podStartE2EDuration="3.793088863s" totalImagesPullingTime="1.306930863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:52.792463722 +0000 UTC m=+98.143111959" watchObservedRunningTime="2026-07-07 19:10:52.793088863 +0000 UTC m=+98.143737101" Jul 07 19:10:53.924039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:53.923991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:53.926331 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:53.926308 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b80486a-4598-4c45-a666-6be9da1b63d9-metrics-tls\") pod \"dns-default-twdjg\" (UID: \"4b80486a-4598-4c45-a666-6be9da1b63d9\") " pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:54.208435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.208357 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g92mm\"" Jul 07 19:10:54.212516 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.212490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:54.216356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.216321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:54.353774 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.353745 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:54.354137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.354116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twdjg"] Jul 07 19:10:54.357236 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:54.357208 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b80486a_4598_4c45_a666_6be9da1b63d9.slice/crio-a70f8a35a9f4d2b0548d88d91c36281ecae23a53fa95b088d9b992e9af1cddba WatchSource:0}: Error finding container a70f8a35a9f4d2b0548d88d91c36281ecae23a53fa95b088d9b992e9af1cddba: Status 404 returned error can't find the container with id a70f8a35a9f4d2b0548d88d91c36281ecae23a53fa95b088d9b992e9af1cddba Jul 07 19:10:54.662336 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.662295 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:54.758658 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.758614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:54.777774 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.777738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twdjg" event={"ID":"4b80486a-4598-4c45-a666-6be9da1b63d9","Type":"ContainerStarted","Data":"a70f8a35a9f4d2b0548d88d91c36281ecae23a53fa95b088d9b992e9af1cddba"} Jul 07 19:10:54.907749 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.907693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:54.957102 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.957069 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ptdb8"] Jul 07 19:10:54.990653 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.990614 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:54.992790 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.992767 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 19:10:54.992962 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.992845 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 19:10:54.993111 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.993097 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzk4r\"" Jul 07 19:10:54.993290 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:54.993277 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 19:10:55.015211 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.015174 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:55.135257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-wtmp\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-root\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-metrics-client-ca\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-sys\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-textfile\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.135491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.135429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2t4\" (UniqueName: \"kubernetes.io/projected/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-kube-api-access-mp2t4\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236157 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2t4\" (UniqueName: \"kubernetes.io/projected/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-kube-api-access-mp2t4\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236157 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236112 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-wtmp\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236157 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-root\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-metrics-client-ca\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236229 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-root\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-sys\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-textfile\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236818 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:55.236639 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 19:10:55.236818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236306 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-wtmp\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.236818 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:55.236697 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls podName:416a3b96-aba2-45b6-a8fd-5ec7cd21db11 nodeName:}" failed. No retries permitted until 2026-07-07 19:10:55.736679358 +0000 UTC m=+101.087327580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls") pod "node-exporter-ptdb8" (UID: "416a3b96-aba2-45b6-a8fd-5ec7cd21db11") : secret "node-exporter-tls" not found Jul 07 19:10:55.236818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236740 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-sys\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.237048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.236887 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.237048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.237018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-textfile\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.237152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.237109 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-metrics-client-ca\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.239476 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.239429 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.258321 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.258237 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2t4\" (UniqueName: \"kubernetes.io/projected/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-kube-api-access-mp2t4\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.302711 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.302671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:55.407890 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.407847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:55.503520 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.503431 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:55.654060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.654021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:55.740806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.740759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.743162 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.743137 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/416a3b96-aba2-45b6-a8fd-5ec7cd21db11-node-exporter-tls\") pod \"node-exporter-ptdb8\" (UID: \"416a3b96-aba2-45b6-a8fd-5ec7cd21db11\") " pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.901898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.901861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptdb8" Jul 07 19:10:55.939837 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:55.939798 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416a3b96_aba2_45b6_a8fd_5ec7cd21db11.slice/crio-6a04daaef7c45d3fb9546b9535d4a156c8c5cf830444a467b3850c5d946909f7 WatchSource:0}: Error finding container 6a04daaef7c45d3fb9546b9535d4a156c8c5cf830444a467b3850c5d946909f7: Status 404 returned error can't find the container with id 6a04daaef7c45d3fb9546b9535d4a156c8c5cf830444a467b3850c5d946909f7 Jul 07 19:10:55.949762 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:55.949733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:56.053435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.053397 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:56.062633 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.062605 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:56.076204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.076165 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.078625 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.078592 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 19:10:56.078775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.078709 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 19:10:56.079668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.079345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 19:10:56.079829 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.079718 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 19:10:56.080500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.080102 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 19:10:56.080500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.080352 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bnccs\"" Jul 07 19:10:56.080500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.080369 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 19:10:56.081393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.081304 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 19:10:56.082480 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.082123 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 19:10:56.082480 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.082425 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 19:10:56.086509 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.086486 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:56.246009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.245899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.245958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.245976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-volume\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556ff\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-kube-api-access-556ff\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246246 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-web-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.246444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.246372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-out\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.347622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.347841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-out\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.347841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.347841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.347841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-volume\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556ff\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-kube-api-access-556ff\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.347996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-web-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348609 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348811 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348744 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.348811 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.348779 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.351167 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.351120 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.351282 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.351248 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-out\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.352089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.352049 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-web-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.352185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.352151 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.352804 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.352706 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-config-volume\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.352804 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.352765 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.352971 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.352874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.353887 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.353863 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.354279 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.354258 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8cdbed3b-5378-4764-af0e-a061df9c7db8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.359326 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.359282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:56.361459 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.361432 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556ff\" (UniqueName: \"kubernetes.io/projected/8cdbed3b-5378-4764-af0e-a061df9c7db8-kube-api-access-556ff\") pod \"alertmanager-main-0\" (UID: \"8cdbed3b-5378-4764-af0e-a061df9c7db8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.395979 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.395938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 19:10:56.787649 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.787613 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptdb8" event={"ID":"416a3b96-aba2-45b6-a8fd-5ec7cd21db11","Type":"ContainerStarted","Data":"6a04daaef7c45d3fb9546b9535d4a156c8c5cf830444a467b3850c5d946909f7"} Jul 07 19:10:56.944715 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:56.944678 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 19:10:57.050511 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:57.050465 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdbed3b_5378_4764_af0e_a061df9c7db8.slice/crio-68b7bd7af3b73adeead424511f4e3041f0b27bb87b28498a98509dd193e3bb3f WatchSource:0}: Error finding container 68b7bd7af3b73adeead424511f4e3041f0b27bb87b28498a98509dd193e3bb3f: Status 404 returned error can't find the container with id 68b7bd7af3b73adeead424511f4e3041f0b27bb87b28498a98509dd193e3bb3f Jul 07 19:10:57.793872 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.793837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twdjg" event={"ID":"4b80486a-4598-4c45-a666-6be9da1b63d9","Type":"ContainerStarted","Data":"ea6bb48538ee1739f2b16ae6064b761365d64f68787feeeed1e5153ce3541511"} Jul 07 19:10:57.794351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.793902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twdjg" event={"ID":"4b80486a-4598-4c45-a666-6be9da1b63d9","Type":"ContainerStarted","Data":"5998f1a8a5f23c98ace3350489cef4d8f8780bcfa8b8d663be1bb5c21539542f"} Jul 07 19:10:57.794351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.793985 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-twdjg" Jul 07 19:10:57.795713 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.795687 2568 generic.go:356] "Generic (PLEG): container finished" podID="416a3b96-aba2-45b6-a8fd-5ec7cd21db11" containerID="d35c4eb0befcc6c362407a90704b60e5bf252a466c1ede31bfc49ad20b52c15a" exitCode=0 Jul 07 19:10:57.795854 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.795792 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptdb8" event={"ID":"416a3b96-aba2-45b6-a8fd-5ec7cd21db11","Type":"ContainerDied","Data":"d35c4eb0befcc6c362407a90704b60e5bf252a466c1ede31bfc49ad20b52c15a"} Jul 07 19:10:57.797273 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.797249 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"68b7bd7af3b73adeead424511f4e3041f0b27bb87b28498a98509dd193e3bb3f"} Jul 07 19:10:57.815841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.815783 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-twdjg" podStartSLOduration=66.3438099 podStartE2EDuration="1m8.815762387s" totalImagesPullingTime="2.471952487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:57.814428571 +0000 UTC m=+103.165076837" watchObservedRunningTime="2026-07-07 19:10:57.815762387 +0000 UTC m=+103.166410623" Jul 07 19:10:57.924556 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:57.924515 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:58.022096 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.022056 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:58.129136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.129102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:58.305943 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.305884 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:58.406238 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.406147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:58.802568 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.802475 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptdb8" event={"ID":"416a3b96-aba2-45b6-a8fd-5ec7cd21db11","Type":"ContainerStarted","Data":"8f4a4482564dedbc431b76af6c933a8ff651293953d76d47e0f9bef20aec9dd4"} Jul 07 19:10:58.802568 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.802547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptdb8" event={"ID":"416a3b96-aba2-45b6-a8fd-5ec7cd21db11","Type":"ContainerStarted","Data":"debce902006a318d97245eabdf320ddd27607719840dd59b49d33934bbb670f4"} Jul 07 19:10:58.803984 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.803955 2568 generic.go:356] "Generic (PLEG): container finished" podID="8cdbed3b-5378-4764-af0e-a061df9c7db8" containerID="3b10f680cc2a9810976026f9511cc40e65cf82a8f7164fb0f3abb37bcf22e0fa" exitCode=0 Jul 07 19:10:58.804105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.804032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerDied","Data":"3b10f680cc2a9810976026f9511cc40e65cf82a8f7164fb0f3abb37bcf22e0fa"} Jul 07 19:10:58.826984 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:58.826907 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ptdb8" podStartSLOduration=3.611653382 podStartE2EDuration="4.826888818s" totalImagesPullingTime="1.135186434s" totalInitContainerRuntime="80.049002ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:58.825741122 +0000 UTC m=+104.176389358" watchObservedRunningTime="2026-07-07 19:10:58.826888818 +0000 UTC m=+104.177537088" Jul 07 19:10:59.418414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.418379 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:59.475388 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.475354 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78d97859f4-lfmk5"] Jul 07 19:10:59.477467 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.477442 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.480652 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.480624 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 19:10:59.480767 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.480629 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 19:10:59.481555 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.481538 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 19:10:59.481555 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.481554 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 19:10:59.481724 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.481603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5346e9sfk9ens\"" Jul 07 19:10:59.481724 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.481705 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6qqgx\"" Jul 07 19:10:59.491451 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.491422 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78d97859f4-lfmk5"] Jul 07 19:10:59.516051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.515996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:59.578868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.578831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.578934 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-client-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.578972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-tls\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.579005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-client-certs\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.579053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmjm\" (UniqueName: \"kubernetes.io/projected/668972cb-27df-40f5-ab4b-26399cc3c110-kube-api-access-pxmjm\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.579155 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/668972cb-27df-40f5-ab4b-26399cc3c110-audit-log\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.579216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.579195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-metrics-server-audit-profiles\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680372 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmjm\" (UniqueName: \"kubernetes.io/projected/668972cb-27df-40f5-ab4b-26399cc3c110-kube-api-access-pxmjm\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680372 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680275 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/668972cb-27df-40f5-ab4b-26399cc3c110-audit-log\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680583 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-metrics-server-audit-profiles\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680583 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680787 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-client-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-tls\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.680889 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.680852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-client-certs\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.683218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.681909 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/668972cb-27df-40f5-ab4b-26399cc3c110-audit-log\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.683218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.682128 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.683218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.682828 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/668972cb-27df-40f5-ab4b-26399cc3c110-metrics-server-audit-profiles\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.688040 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.683942 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-client-certs\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.688040 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.685041 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-client-ca-bundle\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.688040 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.686606 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/668972cb-27df-40f5-ab4b-26399cc3c110-secret-metrics-server-tls\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.715374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.715344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmjm\" (UniqueName: \"kubernetes.io/projected/668972cb-27df-40f5-ab4b-26399cc3c110-kube-api-access-pxmjm\") pod \"metrics-server-78d97859f4-lfmk5\" (UID: \"668972cb-27df-40f5-ab4b-26399cc3c110\") " pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.719795 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.719761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:59.742900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.742864 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5"] Jul 07 19:10:59.745877 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.745852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:10:59.747994 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.747969 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 19:10:59.748123 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.748053 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tvznd\"" Jul 07 19:10:59.758618 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.758550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5"] Jul 07 19:10:59.788190 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.788158 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:10:59.802045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.802017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:10:59.882532 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.882500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkj22\" (UniqueName: \"kubernetes.io/projected/1b2a40ba-6e3c-4569-a372-4c2d74655d15-kube-api-access-tkj22\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:10:59.883002 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.882557 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:10:59.925728 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.925692 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78d97859f4-lfmk5"] Jul 07 19:10:59.930576 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:10:59.930512 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668972cb_27df_40f5_ab4b_26399cc3c110.slice/crio-162182b5a3fa5426d40d59c793757a3a39e6aff6093e45b4030c30894c9b7840 WatchSource:0}: Error finding container 162182b5a3fa5426d40d59c793757a3a39e6aff6093e45b4030c30894c9b7840: Status 404 returned error can't find the container with id 162182b5a3fa5426d40d59c793757a3a39e6aff6093e45b4030c30894c9b7840 Jul 07 19:10:59.983979 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.983940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkj22\" (UniqueName: \"kubernetes.io/projected/1b2a40ba-6e3c-4569-a372-4c2d74655d15-kube-api-access-tkj22\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:10:59.984166 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.984083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:10:59.984475 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:59.984449 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 19:10:59.984601 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:10:59.984518 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert podName:1b2a40ba-6e3c-4569-a372-4c2d74655d15 nodeName:}" failed. No retries permitted until 2026-07-07 19:11:00.484497118 +0000 UTC m=+105.835145340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-9lnx5" (UID: "1b2a40ba-6e3c-4569-a372-4c2d74655d15") : secret "monitoring-plugin-cert" not found Jul 07 19:10:59.993868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:10:59.993838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkj22\" (UniqueName: \"kubernetes.io/projected/1b2a40ba-6e3c-4569-a372-4c2d74655d15-kube-api-access-tkj22\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:00.485292 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:00.485254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:00.489390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:00.489357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:00.489556 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:11:00.489510 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 19:11:00.489600 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:11:00.489576 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert podName:1b2a40ba-6e3c-4569-a372-4c2d74655d15 nodeName:}" failed. No retries permitted until 2026-07-07 19:11:01.489560153 +0000 UTC m=+106.840208372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-9lnx5" (UID: "1b2a40ba-6e3c-4569-a372-4c2d74655d15") : secret "monitoring-plugin-cert" not found Jul 07 19:11:00.584605 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:00.584565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:00.685136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:00.685086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:00.816389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:00.816291 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" event={"ID":"668972cb-27df-40f5-ab4b-26399cc3c110","Type":"ContainerStarted","Data":"162182b5a3fa5426d40d59c793757a3a39e6aff6093e45b4030c30894c9b7840"} Jul 07 19:11:01.326110 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.326076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:01.499579 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.499552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:01.502262 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.502240 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b2a40ba-6e3c-4569-a372-4c2d74655d15-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9lnx5\" (UID: \"1b2a40ba-6e3c-4569-a372-4c2d74655d15\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:01.557682 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.557610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:01.749212 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.749147 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5"] Jul 07 19:11:01.752159 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:11:01.752127 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2a40ba_6e3c_4569_a372_4c2d74655d15.slice/crio-8ab8dd50d547616ed482a85853a89e1d341c00bfdbf4f347381e657ead3b621b WatchSource:0}: Error finding container 8ab8dd50d547616ed482a85853a89e1d341c00bfdbf4f347381e657ead3b621b: Status 404 returned error can't find the container with id 8ab8dd50d547616ed482a85853a89e1d341c00bfdbf4f347381e657ead3b621b Jul 07 19:11:01.823945 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.822095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" event={"ID":"668972cb-27df-40f5-ab4b-26399cc3c110","Type":"ContainerStarted","Data":"e33da1ab16709830a32a233fd9e74c8bea6f108dcf88e9f92d095ffaeffc6be0"} Jul 07 19:11:01.825576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.825545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" event={"ID":"1b2a40ba-6e3c-4569-a372-4c2d74655d15","Type":"ContainerStarted","Data":"8ab8dd50d547616ed482a85853a89e1d341c00bfdbf4f347381e657ead3b621b"} Jul 07 19:11:01.829123 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.829045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"db67e67a3416c9166099a7d14ad05692258fee1978f878db983d4ff8a5a07b93"} Jul 07 19:11:01.829123 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.829110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"8f092adda295e9e4d8ba8f819aeb31446a797ac3ede5c7193a4f41bb4d7d59e7"} Jul 07 19:11:01.829315 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.829137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"44358cb607e362a58b9904532a12209f15a5bbaf23fe6cd56ccbdb3e5632ca87"} Jul 07 19:11:01.829315 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.829162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"67d8d9e646eb8a0de30d7b608e9aff90074e7f0e133a85d1a1849fe43883cf02"} Jul 07 19:11:01.829315 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.829189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"e9f27618697bb10270e5c4deadda4c25c5202ca915ba849a95570f6b64ebfa23"} Jul 07 19:11:01.845737 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:01.845336 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" podStartSLOduration=1.2829014349999999 podStartE2EDuration="2.845314483s" totalImagesPullingTime="1.562413048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:01.843118161 +0000 UTC m=+107.193766410" watchObservedRunningTime="2026-07-07 19:11:01.845314483 +0000 UTC m=+107.195962721" Jul 07 19:11:02.838215 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:02.838170 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8cdbed3b-5378-4764-af0e-a061df9c7db8","Type":"ContainerStarted","Data":"9a263322127b440f941717903f50a2de052febab172fa34934712269ca3c5564"} Jul 07 19:11:02.874857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:02.874802 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.395378885 podStartE2EDuration="6.874784923s" totalImagesPullingTime="4.465341595s" totalInitContainerRuntime="14.064443ms" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:02.873223419 +0000 UTC m=+108.223871717" watchObservedRunningTime="2026-07-07 19:11:02.874784923 +0000 UTC m=+108.225433160" Jul 07 19:11:03.774847 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:03.774790 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:03.843795 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:03.843750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" event={"ID":"1b2a40ba-6e3c-4569-a372-4c2d74655d15","Type":"ContainerStarted","Data":"3d89bae0cba63ccb19f62f290347c187a56c7d4969756f41e17eda2bb00c6ab3"} Jul 07 19:11:03.844317 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:03.844281 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:03.849123 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:03.849098 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" Jul 07 19:11:03.862329 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:03.862276 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9lnx5" podStartSLOduration=3.326938413 podStartE2EDuration="4.862258647s" totalImagesPullingTime="1.535320234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:03.860613529 +0000 UTC m=+109.211261767" watchObservedRunningTime="2026-07-07 19:11:03.862258647 +0000 UTC m=+109.212906884" Jul 07 19:11:04.743315 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:04.743286 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f85858d46-w926r" Jul 07 19:11:04.895173 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:04.895138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:04.995771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:04.995673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:06.639999 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.639942 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" podUID="61f376af-e5c7-48df-888c-1a82f9581889" containerName="registry" containerID="cri-o://7990431ebc74bb0c241c3e036487640cf79896b43a4a71f05d2d4f25be5786a5" gracePeriod=30 Jul 07 19:11:06.859848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.859817 2568 generic.go:356] "Generic (PLEG): container finished" podID="61f376af-e5c7-48df-888c-1a82f9581889" containerID="7990431ebc74bb0c241c3e036487640cf79896b43a4a71f05d2d4f25be5786a5" exitCode=0 Jul 07 19:11:06.860039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.859883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" event={"ID":"61f376af-e5c7-48df-888c-1a82f9581889","Type":"ContainerDied","Data":"7990431ebc74bb0c241c3e036487640cf79896b43a4a71f05d2d4f25be5786a5"} Jul 07 19:11:06.891976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.891892 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:11:06.947120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947076 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947118 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947134 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947161 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947183 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dpkt\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947353 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947610 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947384 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration\") pod \"61f376af-e5c7-48df-888c-1a82f9581889\" (UID: \"61f376af-e5c7-48df-888c-1a82f9581889\") " Jul 07 19:11:06.947678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947628 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:11:06.947735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.947715 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:11:06.949863 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.949830 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:11:06.950035 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.949931 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:11:06.950102 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.950055 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:11:06.950162 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.950108 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt" (OuterVolumeSpecName: "kube-api-access-9dpkt") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "kube-api-access-9dpkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:11:06.950162 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.950117 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:11:06.955852 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:06.955822 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "61f376af-e5c7-48df-888c-1a82f9581889" (UID: "61f376af-e5c7-48df-888c-1a82f9581889"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:11:07.048883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048828 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-image-registry-private-configuration\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.048883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048876 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/61f376af-e5c7-48df-888c-1a82f9581889-installation-pull-secrets\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.048883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048887 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-trusted-ca\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.048883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048895 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-bound-sa-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.048883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048904 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/61f376af-e5c7-48df-888c-1a82f9581889-registry-certificates\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.049204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048912 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-registry-tls\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.049204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048942 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpkt\" (UniqueName: \"kubernetes.io/projected/61f376af-e5c7-48df-888c-1a82f9581889-kube-api-access-9dpkt\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.049204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.048951 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/61f376af-e5c7-48df-888c-1a82f9581889-ca-trust-extracted\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:11:07.806469 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.806441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-twdjg" Jul 07 19:11:07.868659 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.868631 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" Jul 07 19:11:07.868850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.868626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6749c54d-cm8gn" event={"ID":"61f376af-e5c7-48df-888c-1a82f9581889","Type":"ContainerDied","Data":"6e1b298a22e4b9afe3cb52b3f523744827ded0d87bbae807ab7f249b4efa8f3e"} Jul 07 19:11:07.868850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.868774 2568 scope.go:117] "RemoveContainer" containerID="7990431ebc74bb0c241c3e036487640cf79896b43a4a71f05d2d4f25be5786a5" Jul 07 19:11:07.886070 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.886037 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:11:07.892604 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:07.892573 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c6749c54d-cm8gn"] Jul 07 19:11:09.352362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:09.352327 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f376af-e5c7-48df-888c-1a82f9581889" path="/var/lib/kubelet/pods/61f376af-e5c7-48df-888c-1a82f9581889/volumes" Jul 07 19:11:11.328930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:11.328879 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:11.430133 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:11.430099 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:11.497407 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:11.497375 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:12.173380 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:12.173346 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:12.270637 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:12.270601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:13.164731 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:13.164697 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:13.236752 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:13.236713 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:19.788633 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:19.788581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:11:19.789169 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:19.788660 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:11:20.248546 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:20.248500 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:20.351850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:20.351817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:22.153019 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:22.152979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:22.216927 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:22.216876 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:22.321012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:22.320974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:22.391319 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:22.391284 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:25.014021 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.013987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:11:25.016413 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.016390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8120d1f0-2e6a-40e7-8017-7923fce34250-metrics-certs\") pod \"network-metrics-daemon-lk246\" (UID: \"8120d1f0-2e6a-40e7-8017-7923fce34250\") " pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:11:25.169836 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.169805 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jqhn7\"" Jul 07 19:11:25.178581 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.178554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk246" Jul 07 19:11:25.353934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.353849 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lk246"] Jul 07 19:11:25.356567 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:11:25.356534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8120d1f0_2e6a_40e7_8017_7923fce34250.slice/crio-88053aadade636b0f7a2e67b54bdcf0266c4a36dcc1454a05c89275e8f7c3aaf WatchSource:0}: Error finding container 88053aadade636b0f7a2e67b54bdcf0266c4a36dcc1454a05c89275e8f7c3aaf: Status 404 returned error can't find the container with id 88053aadade636b0f7a2e67b54bdcf0266c4a36dcc1454a05c89275e8f7c3aaf Jul 07 19:11:25.937459 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:25.937427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk246" event={"ID":"8120d1f0-2e6a-40e7-8017-7923fce34250","Type":"ContainerStarted","Data":"88053aadade636b0f7a2e67b54bdcf0266c4a36dcc1454a05c89275e8f7c3aaf"} Jul 07 19:11:26.944026 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:26.943983 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk246" event={"ID":"8120d1f0-2e6a-40e7-8017-7923fce34250","Type":"ContainerStarted","Data":"6a0088cd55dca1bdb60c4a4288f7c01df3601bc6104c5303743f61d3c767cd59"} Jul 07 19:11:26.944468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:26.944074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk246" event={"ID":"8120d1f0-2e6a-40e7-8017-7923fce34250","Type":"ContainerStarted","Data":"4df45b9698a7cede36eb35ca4fd1c44ae355adee9c19bd069be7eaea61973949"} Jul 07 19:11:26.963266 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:26.963213 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lk246" podStartSLOduration=130.756998292 podStartE2EDuration="2m11.963196136s" totalImagesPullingTime="1.206197844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:26.962464703 +0000 UTC m=+132.313112940" watchObservedRunningTime="2026-07-07 19:11:26.963196136 +0000 UTC m=+132.313844373" Jul 07 19:11:27.080830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.080777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.164332 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.164300 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.246011 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.245886 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.506719 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.506624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.621374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.621337 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.704522 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.704489 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.813074 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.812992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:27.881365 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:27.881334 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:34.977833 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:34.977742 2568 generic.go:356] "Generic (PLEG): container finished" podID="0a587bbe-ddfc-4ebd-970c-f3b2795ae36c" containerID="2d9edbdaf3509eadae04415f1ef3e93c7e151cd4fc54b3615b541991f7cc1f12" exitCode=0 Jul 07 19:11:34.977833 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:34.977815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" event={"ID":"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c","Type":"ContainerDied","Data":"2d9edbdaf3509eadae04415f1ef3e93c7e151cd4fc54b3615b541991f7cc1f12"} Jul 07 19:11:34.978328 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:34.978183 2568 scope.go:117] "RemoveContainer" containerID="2d9edbdaf3509eadae04415f1ef3e93c7e151cd4fc54b3615b541991f7cc1f12" Jul 07 19:11:35.258990 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:35.258877 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:35.359735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:35.359695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:35.983066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:35.983033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-x44b6" event={"ID":"0a587bbe-ddfc-4ebd-970c-f3b2795ae36c","Type":"ContainerStarted","Data":"f8c26a5a0ec253dc0d2a2f7409486c6d164cb6b7c9228eac97bb4843352a75d6"} Jul 07 19:11:39.793751 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:39.793718 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:11:39.797600 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:39.797571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78d97859f4-lfmk5" Jul 07 19:11:43.403981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:43.403946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:48.029496 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:48.029463 2568 generic.go:356] "Generic (PLEG): container finished" podID="a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758" containerID="cfc92b3a54a5f496706400de1562d6f8ff8bf7855885d28e986f3da1dcb33392" exitCode=0 Jul 07 19:11:48.029878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:48.029539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" event={"ID":"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758","Type":"ContainerDied","Data":"cfc92b3a54a5f496706400de1562d6f8ff8bf7855885d28e986f3da1dcb33392"} Jul 07 19:11:48.029970 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:48.029955 2568 scope.go:117] "RemoveContainer" containerID="cfc92b3a54a5f496706400de1562d6f8ff8bf7855885d28e986f3da1dcb33392" Jul 07 19:11:48.816538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:48.816503 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:48.888459 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:48.888425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:11:49.035149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:49.035117 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-b2pvv" event={"ID":"a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758","Type":"ContainerStarted","Data":"dbaaa2fda2bce46da88116d2578825ce5073fd251f7c1472b36339404c39fbe6"} Jul 07 19:11:50.578410 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:11:50.578376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:14.297071 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:14.296987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:14.499619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:14.499587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:19.411362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.411327 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74bb8587-gpsnc"] Jul 07 19:12:19.412295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.412272 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f376af-e5c7-48df-888c-1a82f9581889" containerName="registry" Jul 07 19:12:19.412440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.412429 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f376af-e5c7-48df-888c-1a82f9581889" containerName="registry" Jul 07 19:12:19.412621 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.412610 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f376af-e5c7-48df-888c-1a82f9581889" containerName="registry" Jul 07 19:12:19.415792 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.415772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.418060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418034 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 19:12:19.418060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418051 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 19:12:19.418275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 19:12:19.418275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418077 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 19:12:19.418275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418137 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 19:12:19.418275 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.418088 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-79pdm\"" Jul 07 19:12:19.423197 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.423064 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 19:12:19.428602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.428575 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74bb8587-gpsnc"] Jul 07 19:12:19.498200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-metrics-client-ca\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-federate-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-serving-certs-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.498609 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.498452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mzw\" (UniqueName: \"kubernetes.io/projected/df8468d8-2ba6-4cb0-b571-919ae29116f7-kube-api-access-84mzw\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.599755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.599708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.599755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.599764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-serving-certs-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.599881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mzw\" (UniqueName: \"kubernetes.io/projected/df8468d8-2ba6-4cb0-b571-919ae29116f7-kube-api-access-84mzw\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.599987 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-metrics-client-ca\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-federate-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600308 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600643 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-serving-certs-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.600802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.600671 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-metrics-client-ca\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.601243 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.601215 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.602590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.602557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-telemeter-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.602590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.602589 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.602743 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.602597 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-secret-telemeter-client\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.602743 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.602623 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df8468d8-2ba6-4cb0-b571-919ae29116f7-federate-client-tls\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.610233 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.610205 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mzw\" (UniqueName: \"kubernetes.io/projected/df8468d8-2ba6-4cb0-b571-919ae29116f7-kube-api-access-84mzw\") pod \"telemeter-client-74bb8587-gpsnc\" (UID: \"df8468d8-2ba6-4cb0-b571-919ae29116f7\") " pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.727019 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.726901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" Jul 07 19:12:19.866238 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:19.866211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74bb8587-gpsnc"] Jul 07 19:12:19.873257 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:12:19.868259 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8468d8_2ba6_4cb0_b571_919ae29116f7.slice/crio-ac87f0c14ac79b09b3be2a54062e69b9e27179e7b00cfbb7d0530b645e152eb1 WatchSource:0}: Error finding container ac87f0c14ac79b09b3be2a54062e69b9e27179e7b00cfbb7d0530b645e152eb1: Status 404 returned error can't find the container with id ac87f0c14ac79b09b3be2a54062e69b9e27179e7b00cfbb7d0530b645e152eb1 Jul 07 19:12:20.162381 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:20.162346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" event={"ID":"df8468d8-2ba6-4cb0-b571-919ae29116f7","Type":"ContainerStarted","Data":"ac87f0c14ac79b09b3be2a54062e69b9e27179e7b00cfbb7d0530b645e152eb1"} Jul 07 19:12:22.172081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:22.172047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" event={"ID":"df8468d8-2ba6-4cb0-b571-919ae29116f7","Type":"ContainerStarted","Data":"09955b1a1e2be5edd7a6f42a6bd66d6ed593dcb900c0825f635a67807d047600"} Jul 07 19:12:23.177648 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:23.177607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" event={"ID":"df8468d8-2ba6-4cb0-b571-919ae29116f7","Type":"ContainerStarted","Data":"e5ad09443c08413be02dec804192e04880caeb6e12446de95962f6165d086fb2"} Jul 07 19:12:23.178139 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:23.177671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" event={"ID":"df8468d8-2ba6-4cb0-b571-919ae29116f7","Type":"ContainerStarted","Data":"e58116dbf9507345b14f6b16228714a7649f2c7ffc082de8f89f2a591181aae6"} Jul 07 19:12:23.201335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:23.201268 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74bb8587-gpsnc" podStartSLOduration=1.9738321490000001 podStartE2EDuration="4.201252921s" totalImagesPullingTime="2.227420772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:12:23.200992568 +0000 UTC m=+188.551640831" watchObservedRunningTime="2026-07-07 19:12:23.201252921 +0000 UTC m=+188.551901156" Jul 07 19:12:23.897274 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:23.897241 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:29.577721 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:29.577681 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:29.636565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:29.636518 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:41.460838 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:41.460800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:41.530155 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:41.530119 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:44.697201 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:44.697169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:44.790129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:44.790099 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:44.849883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:44.849840 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:50.237520 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:50.237482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:12:50.337630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:12:50.337591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:18.957551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:18.957518 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:30.201700 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:30.201650 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:42.083969 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.083860 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:42.189057 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.189021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:13:42.709857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.709819 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r"] Jul 07 19:13:42.717117 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.717085 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.719712 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.719690 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:13:42.720410 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.720383 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:13:42.720536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.720396 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kws22\"" Jul 07 19:13:42.722571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.722545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r"] Jul 07 19:13:42.837383 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.837343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.837561 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.837410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.837561 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.837431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ww4t\" (UniqueName: \"kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.938245 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.938210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.938433 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.938269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.938433 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.938375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ww4t\" (UniqueName: \"kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.938619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.938598 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.938683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.938625 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:42.948639 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:42.948602 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ww4t\" (UniqueName: \"kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:43.028517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:43.028416 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:13:43.167039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:43.167014 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r"] Jul 07 19:13:43.169722 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:13:43.169687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa12ea9c_bb56_4e6a_926e_578f41d61c4e.slice/crio-bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5 WatchSource:0}: Error finding container bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5: Status 404 returned error can't find the container with id bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5 Jul 07 19:13:43.488597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:43.488557 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerStarted","Data":"bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5"} Jul 07 19:13:51.524894 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:51.524860 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerID="666fce36e726d5c4ab6bfd88585ac387c239892401b080231f7900bd4bea154f" exitCode=0 Jul 07 19:13:51.525324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:51.524909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerDied","Data":"666fce36e726d5c4ab6bfd88585ac387c239892401b080231f7900bd4bea154f"} Jul 07 19:13:53.537094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:53.537052 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerStarted","Data":"90def39c4dbd43a13cf568ed55dd956e40150780993bfc84242bfc94c15aa01a"} Jul 07 19:13:54.542980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:54.542946 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerID="90def39c4dbd43a13cf568ed55dd956e40150780993bfc84242bfc94c15aa01a" exitCode=0 Jul 07 19:13:54.543480 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:13:54.543016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerDied","Data":"90def39c4dbd43a13cf568ed55dd956e40150780993bfc84242bfc94c15aa01a"} Jul 07 19:14:05.590356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:05.590322 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerID="ed1aa252ff2f6adb5bce82cdd3ad21de97fed4f472311786a0ce9409a5ad4972" exitCode=0 Jul 07 19:14:05.590764 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:05.590393 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerDied","Data":"ed1aa252ff2f6adb5bce82cdd3ad21de97fed4f472311786a0ce9409a5ad4972"} Jul 07 19:14:06.750736 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.750702 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:14:06.845871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.845776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util\") pod \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " Jul 07 19:14:06.845871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.845820 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle\") pod \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " Jul 07 19:14:06.846081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.845875 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ww4t\" (UniqueName: \"kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t\") pod \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\" (UID: \"aa12ea9c-bb56-4e6a-926e-578f41d61c4e\") " Jul 07 19:14:06.846546 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.846517 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle" (OuterVolumeSpecName: "bundle") pod "aa12ea9c-bb56-4e6a-926e-578f41d61c4e" (UID: "aa12ea9c-bb56-4e6a-926e-578f41d61c4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:06.848196 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.848166 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t" (OuterVolumeSpecName: "kube-api-access-7ww4t") pod "aa12ea9c-bb56-4e6a-926e-578f41d61c4e" (UID: "aa12ea9c-bb56-4e6a-926e-578f41d61c4e"). InnerVolumeSpecName "kube-api-access-7ww4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:06.850602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.850576 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util" (OuterVolumeSpecName: "util") pod "aa12ea9c-bb56-4e6a-926e-578f41d61c4e" (UID: "aa12ea9c-bb56-4e6a-926e-578f41d61c4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:06.947487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.947444 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:06.947487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.947479 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:06.947487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:06.947489 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ww4t\" (UniqueName: \"kubernetes.io/projected/aa12ea9c-bb56-4e6a-926e-578f41d61c4e-kube-api-access-7ww4t\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:07.600220 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:07.600183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" event={"ID":"aa12ea9c-bb56-4e6a-926e-578f41d61c4e","Type":"ContainerDied","Data":"bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5"} Jul 07 19:14:07.600393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:07.600227 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754042v9r" Jul 07 19:14:07.600393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:07.600240 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe513c3671d432ceba59c11722fa4fd95092f95e7a972c2c547522fb707abe5" Jul 07 19:14:08.809579 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:08.809547 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:09.628615 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:09.628562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:12.610235 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:12.610201 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:12.674454 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:12.674413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:14.295451 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:14.295412 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:14.401117 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:14.401091 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:14.654777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:14.654740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:14.766177 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:14.766143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:15.143144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.143115 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 19:14:15.245736 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.245697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46"] Jul 07 19:14:15.246080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246064 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="extract" Jul 07 19:14:15.246134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246082 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="extract" Jul 07 19:14:15.246134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246105 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="util" Jul 07 19:14:15.246134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246110 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="util" Jul 07 19:14:15.246134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246117 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="pull" Jul 07 19:14:15.246134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246122 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="pull" Jul 07 19:14:15.246280 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.246185 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa12ea9c-bb56-4e6a-926e-578f41d61c4e" containerName="extract" Jul 07 19:14:15.248492 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.248472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.251219 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.251186 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:14:15.251525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.251505 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:14:15.251994 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.251978 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kws22\"" Jul 07 19:14:15.261980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.261909 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46"] Jul 07 19:14:15.314304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.314143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w7m\" (UniqueName: \"kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.314304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.314196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.314304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.314275 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.414991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.414870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.414991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.414972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w7m\" (UniqueName: \"kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.415181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.415013 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.415356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.415340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.415396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.415336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.429945 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.429903 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w7m\" (UniqueName: \"kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.557970 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.557907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:15.699817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:15.699729 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46"] Jul 07 19:14:15.703249 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:14:15.703223 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590c0a0d_bdb8_4c16_aa20_6ce6dfc730dd.slice/crio-4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585 WatchSource:0}: Error finding container 4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585: Status 404 returned error can't find the container with id 4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585 Jul 07 19:14:16.525745 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:16.525708 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:16.640020 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:16.639984 2568 generic.go:356] "Generic (PLEG): container finished" podID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerID="eb38a00f8d0bad00d57591b55b5b1a589b14ab3b9e02ad2968a9d127e4b69a54" exitCode=0 Jul 07 19:14:16.640216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:16.640075 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" event={"ID":"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd","Type":"ContainerDied","Data":"eb38a00f8d0bad00d57591b55b5b1a589b14ab3b9e02ad2968a9d127e4b69a54"} Jul 07 19:14:16.640216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:16.640132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" event={"ID":"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd","Type":"ContainerStarted","Data":"4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585"} Jul 07 19:14:16.641225 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:16.641208 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:14:17.101203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.100963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn"] Jul 07 19:14:17.103783 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.103752 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.106290 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.106259 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 19:14:17.106290 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.106282 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 19:14:17.114815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.114775 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn"] Jul 07 19:14:17.129909 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.129870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pbd\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-kube-api-access-w5pbd\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.130112 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.129964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.158492 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.158410 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:17.231683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.231650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pbd\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-kube-api-access-w5pbd\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.231890 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.231707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.241224 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.241189 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.241399 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.241291 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pbd\" (UniqueName: \"kubernetes.io/projected/60e8c5f8-5303-4cde-994b-c429a853bd78-kube-api-access-w5pbd\") pod \"cert-manager-cainjector-559cdd8c87-ggmmn\" (UID: \"60e8c5f8-5303-4cde-994b-c429a853bd78\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.417834 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.417794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" Jul 07 19:14:17.584998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.584969 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn"] Jul 07 19:14:17.758834 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.758743 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:17.859755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:17.859710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:18.285316 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:14:18.285266 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e8c5f8_5303_4cde_994b_c429a853bd78.slice/crio-8112f3b2b82df9fd3518feb25c84dd845b9bc5fed3651abe18180583f463f12a WatchSource:0}: Error finding container 8112f3b2b82df9fd3518feb25c84dd845b9bc5fed3651abe18180583f463f12a: Status 404 returned error can't find the container with id 8112f3b2b82df9fd3518feb25c84dd845b9bc5fed3651abe18180583f463f12a Jul 07 19:14:18.650510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:18.650477 2568 generic.go:356] "Generic (PLEG): container finished" podID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerID="89d6755f19665d738d010aec871fbc8380697ee9ffd5765dcc1326b8a105e9f4" exitCode=0 Jul 07 19:14:18.650969 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:18.650569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" event={"ID":"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd","Type":"ContainerDied","Data":"89d6755f19665d738d010aec871fbc8380697ee9ffd5765dcc1326b8a105e9f4"} Jul 07 19:14:18.651839 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:18.651813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" event={"ID":"60e8c5f8-5303-4cde-994b-c429a853bd78","Type":"ContainerStarted","Data":"8112f3b2b82df9fd3518feb25c84dd845b9bc5fed3651abe18180583f463f12a"} Jul 07 19:14:19.212567 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:19.212531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:19.313497 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:19.313446 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:19.659299 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:19.659192 2568 generic.go:356] "Generic (PLEG): container finished" podID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerID="50fc0827b28e10a5fb6163b96df6b960502f9f8acb33e648069ac4cfae3d87fb" exitCode=0 Jul 07 19:14:19.659299 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:19.659286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" event={"ID":"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd","Type":"ContainerDied","Data":"50fc0827b28e10a5fb6163b96df6b960502f9f8acb33e648069ac4cfae3d87fb"} Jul 07 19:14:20.850840 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.850815 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:20.966091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.966059 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util\") pod \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " Jul 07 19:14:20.966263 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.966131 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle\") pod \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " Jul 07 19:14:20.966263 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.966166 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6w7m\" (UniqueName: \"kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m\") pod \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\" (UID: \"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd\") " Jul 07 19:14:20.966531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.966509 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle" (OuterVolumeSpecName: "bundle") pod "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" (UID: "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:20.968322 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.968297 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m" (OuterVolumeSpecName: "kube-api-access-q6w7m") pod "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" (UID: "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd"). InnerVolumeSpecName "kube-api-access-q6w7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:20.971905 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:20.971872 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util" (OuterVolumeSpecName: "util") pod "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" (UID: "590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:21.066882 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.066843 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:21.066882 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.066876 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:21.066882 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.066889 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6w7m\" (UniqueName: \"kubernetes.io/projected/590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd-kube-api-access-q6w7m\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:21.670864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.670767 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" Jul 07 19:14:21.670864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.670771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2rn46" event={"ID":"590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd","Type":"ContainerDied","Data":"4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585"} Jul 07 19:14:21.671088 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.670898 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f87bd27d98a99f8175c0a899ff1087e5f87f74c00259f6c51de833fabb1b585" Jul 07 19:14:21.672511 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.672474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" event={"ID":"60e8c5f8-5303-4cde-994b-c429a853bd78","Type":"ContainerStarted","Data":"de09f64c021997254435480edee4e56e24ec63d7e1f7ee413986eac86d0672d2"} Jul 07 19:14:21.692505 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:21.692452 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ggmmn" podStartSLOduration=2.10719431 podStartE2EDuration="4.692434464s" totalImagesPullingTime="2.585240154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:21.690970408 +0000 UTC m=+307.041618653" watchObservedRunningTime="2026-07-07 19:14:21.692434464 +0000 UTC m=+307.043082700" Jul 07 19:14:22.095616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:22.095518 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:24.976933 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:24.976893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:29.611285 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:29.611249 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:29.711466 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:29.711406 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:31.855486 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855447 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h"] Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855760 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="pull" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855771 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="pull" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855781 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="util" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855786 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="util" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855795 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="extract" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855801 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="extract" Jul 07 19:14:31.855851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.855851 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="590c0a0d-bdb8-4c16-aa20-6ce6dfc730dd" containerName="extract" Jul 07 19:14:31.858005 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.857989 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:31.860445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.860416 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:14:31.860586 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.860534 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d5dk9\"" Jul 07 19:14:31.861114 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.861094 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 19:14:31.870960 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.870930 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h"] Jul 07 19:14:31.968284 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.968250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b129705-0072-40fe-9af9-6bb49eef1167-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:31.968474 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:31.968310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwmb\" (UniqueName: \"kubernetes.io/projected/6b129705-0072-40fe-9af9-6bb49eef1167-kube-api-access-vrwmb\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.069645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.069612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b129705-0072-40fe-9af9-6bb49eef1167-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.069843 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.069662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwmb\" (UniqueName: \"kubernetes.io/projected/6b129705-0072-40fe-9af9-6bb49eef1167-kube-api-access-vrwmb\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.070084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.070064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b129705-0072-40fe-9af9-6bb49eef1167-tmp\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.082088 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.082049 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwmb\" (UniqueName: \"kubernetes.io/projected/6b129705-0072-40fe-9af9-6bb49eef1167-kube-api-access-vrwmb\") pod \"openshift-lws-operator-fd8ccff4c-t4f8h\" (UID: \"6b129705-0072-40fe-9af9-6bb49eef1167\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.166885 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.166844 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" Jul 07 19:14:32.305043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.305016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h"] Jul 07 19:14:32.307816 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:14:32.307786 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b129705_0072_40fe_9af9_6bb49eef1167.slice/crio-0ff35da0c85b55662eb9cb13cd40b9f1e854ef652fff50c4805b5a757f201055 WatchSource:0}: Error finding container 0ff35da0c85b55662eb9cb13cd40b9f1e854ef652fff50c4805b5a757f201055: Status 404 returned error can't find the container with id 0ff35da0c85b55662eb9cb13cd40b9f1e854ef652fff50c4805b5a757f201055 Jul 07 19:14:32.726048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:32.726017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" event={"ID":"6b129705-0072-40fe-9af9-6bb49eef1167","Type":"ContainerStarted","Data":"0ff35da0c85b55662eb9cb13cd40b9f1e854ef652fff50c4805b5a757f201055"} Jul 07 19:14:34.736850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:34.736815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" event={"ID":"6b129705-0072-40fe-9af9-6bb49eef1167","Type":"ContainerStarted","Data":"b707e195ba5d2f304100b652d5fff8703d4437bb264808faaf56d50c9169a151"} Jul 07 19:14:34.762812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:34.762754 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-t4f8h" podStartSLOduration=1.760365761 podStartE2EDuration="3.762731578s" totalImagesPullingTime="2.002365817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:34.760880213 +0000 UTC m=+320.111528454" watchObservedRunningTime="2026-07-07 19:14:34.762731578 +0000 UTC m=+320.113379815" Jul 07 19:14:35.777233 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:35.777192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:36.982136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:36.982095 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:37.728873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:37.728830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:39.093335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.093289 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n"] Jul 07 19:14:39.095746 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.095727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.098495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.098467 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:14:39.098644 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.098508 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:14:39.099022 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.099005 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kws22\"" Jul 07 19:14:39.107227 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.107195 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n"] Jul 07 19:14:39.237052 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.237010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.237052 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.237046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.237262 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.237130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2cxl\" (UniqueName: \"kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.338391 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.338362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.338530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.338396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.338530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.338462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2cxl\" (UniqueName: \"kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.338862 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.338836 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.338898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.338842 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.348730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.348652 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2cxl\" (UniqueName: \"kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.406558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.406507 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:39.558897 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.558863 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n"] Jul 07 19:14:39.562229 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:14:39.562189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7c0086_ce4c_466e_b1b8_4aebb9235d69.slice/crio-6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb WatchSource:0}: Error finding container 6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb: Status 404 returned error can't find the container with id 6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb Jul 07 19:14:39.761536 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.761499 2568 generic.go:356] "Generic (PLEG): container finished" podID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerID="6532c1b04434a02f7e771aff51156df918e92aec8a7a82f21969fac8280ddd11" exitCode=0 Jul 07 19:14:39.761709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.761577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" event={"ID":"1f7c0086-ce4c-466e-b1b8-4aebb9235d69","Type":"ContainerDied","Data":"6532c1b04434a02f7e771aff51156df918e92aec8a7a82f21969fac8280ddd11"} Jul 07 19:14:39.761709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:39.761628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" event={"ID":"1f7c0086-ce4c-466e-b1b8-4aebb9235d69","Type":"ContainerStarted","Data":"6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb"} Jul 07 19:14:40.767767 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:40.767736 2568 generic.go:356] "Generic (PLEG): container finished" podID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerID="7e2af5011d82249f3dbd663b5d51d6878c7b187be78447df71e9d58adaa613f5" exitCode=0 Jul 07 19:14:40.768240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:40.767820 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" event={"ID":"1f7c0086-ce4c-466e-b1b8-4aebb9235d69","Type":"ContainerDied","Data":"7e2af5011d82249f3dbd663b5d51d6878c7b187be78447df71e9d58adaa613f5"} Jul 07 19:14:41.775419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:41.775372 2568 generic.go:356] "Generic (PLEG): container finished" podID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerID="958d5149c889602412a00b2e8affe792c78fdd17d01762f7054d94f8a4b60298" exitCode=0 Jul 07 19:14:41.775827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:41.775507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" event={"ID":"1f7c0086-ce4c-466e-b1b8-4aebb9235d69","Type":"ContainerDied","Data":"958d5149c889602412a00b2e8affe792c78fdd17d01762f7054d94f8a4b60298"} Jul 07 19:14:42.915240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:42.915212 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:43.069905 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.069811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2cxl\" (UniqueName: \"kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl\") pod \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " Jul 07 19:14:43.069905 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.069873 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle\") pod \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " Jul 07 19:14:43.070129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.069949 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util\") pod \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\" (UID: \"1f7c0086-ce4c-466e-b1b8-4aebb9235d69\") " Jul 07 19:14:43.070671 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.070640 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle" (OuterVolumeSpecName: "bundle") pod "1f7c0086-ce4c-466e-b1b8-4aebb9235d69" (UID: "1f7c0086-ce4c-466e-b1b8-4aebb9235d69"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:43.072108 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.072079 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl" (OuterVolumeSpecName: "kube-api-access-t2cxl") pod "1f7c0086-ce4c-466e-b1b8-4aebb9235d69" (UID: "1f7c0086-ce4c-466e-b1b8-4aebb9235d69"). InnerVolumeSpecName "kube-api-access-t2cxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:14:43.075777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.075748 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util" (OuterVolumeSpecName: "util") pod "1f7c0086-ce4c-466e-b1b8-4aebb9235d69" (UID: "1f7c0086-ce4c-466e-b1b8-4aebb9235d69"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:14:43.170788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.170745 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2cxl\" (UniqueName: \"kubernetes.io/projected/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-kube-api-access-t2cxl\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:43.170788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.170780 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:43.170788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.170790 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f7c0086-ce4c-466e-b1b8-4aebb9235d69-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:14:43.787551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.787507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" event={"ID":"1f7c0086-ce4c-466e-b1b8-4aebb9235d69","Type":"ContainerDied","Data":"6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb"} Jul 07 19:14:43.787551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.787534 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6kr2n" Jul 07 19:14:43.787551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:43.787560 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a594214de039e4bfc64922a30fbb93d5933df1bdb278e189a12abe8217173bb" Jul 07 19:14:50.838662 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:50.838623 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:50.976326 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:50.976279 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.447955 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.447905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.716331 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.716236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.799157 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799119 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw"] Jul 07 19:14:51.799454 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799441 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="pull" Jul 07 19:14:51.799548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799456 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="pull" Jul 07 19:14:51.799548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799473 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="util" Jul 07 19:14:51.799548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799478 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="util" Jul 07 19:14:51.799548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799487 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="extract" Jul 07 19:14:51.799548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799492 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="extract" Jul 07 19:14:51.799697 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.799555 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7c0086-ce4c-466e-b1b8-4aebb9235d69" containerName="extract" Jul 07 19:14:51.802440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.802416 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.805059 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.805036 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 19:14:51.805189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.805036 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 19:14:51.805506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.805487 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 19:14:51.805733 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.805708 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 19:14:51.805881 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.805866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2ttzd\"" Jul 07 19:14:51.820215 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.819515 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw"] Jul 07 19:14:51.822542 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.822500 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:51.849396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.849358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wdp\" (UniqueName: \"kubernetes.io/projected/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-kube-api-access-t8wdp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.849396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.849400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-manifests\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.849844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.849437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.849844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.849554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-tmp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.849844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.849582 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-webhook-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.950431 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.950386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-tmp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.950431 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.950435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-webhook-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.950684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.950487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wdp\" (UniqueName: \"kubernetes.io/projected/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-kube-api-access-t8wdp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.950684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.950514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-manifests\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.950684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.950549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.951031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.951008 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-manifests\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.952667 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.952643 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-tmp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.952890 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.952873 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-webhook-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.953062 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.953044 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.965817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.965782 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wdp\" (UniqueName: \"kubernetes.io/projected/8e1eba71-1c5d-457e-8878-7b6a3fb5ff56-kube-api-access-t8wdp\") pod \"opendatahub-operator-controller-manager-75847b84df-hccdw\" (UID: \"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56\") " pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:51.995855 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:51.995771 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.113501 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.113460 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:52.170654 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.170604 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.285434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.282988 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw"] Jul 07 19:14:52.286861 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:14:52.286818 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1eba71_1c5d_457e_8878_7b6a3fb5ff56.slice/crio-aa93b0d59d5c673c5516d857a82b6d1949cd47f00d39d27d32df7e5da8b37c28 WatchSource:0}: Error finding container aa93b0d59d5c673c5516d857a82b6d1949cd47f00d39d27d32df7e5da8b37c28: Status 404 returned error can't find the container with id aa93b0d59d5c673c5516d857a82b6d1949cd47f00d39d27d32df7e5da8b37c28 Jul 07 19:14:52.299650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.299621 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.412672 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.412638 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.510618 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.510579 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.619087 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.619054 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:52.832739 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:52.832686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" event={"ID":"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56","Type":"ContainerStarted","Data":"aa93b0d59d5c673c5516d857a82b6d1949cd47f00d39d27d32df7e5da8b37c28"} Jul 07 19:14:54.348074 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:54.348040 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:54.437937 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:54.437885 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:14:55.854770 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:55.854732 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" event={"ID":"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56","Type":"ContainerStarted","Data":"3ff6a2433f7f6c822149c043d5fb8af81eb50262e0adc8e5de6b380d6ac7d859"} Jul 07 19:14:56.860610 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:56.860577 2568 generic.go:356] "Generic (PLEG): container finished" podID="8e1eba71-1c5d-457e-8878-7b6a3fb5ff56" containerID="3ff6a2433f7f6c822149c043d5fb8af81eb50262e0adc8e5de6b380d6ac7d859" exitCode=0 Jul 07 19:14:56.861012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:56.860638 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" event={"ID":"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56","Type":"ContainerDied","Data":"3ff6a2433f7f6c822149c043d5fb8af81eb50262e0adc8e5de6b380d6ac7d859"} Jul 07 19:14:57.866893 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:57.866857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" event={"ID":"8e1eba71-1c5d-457e-8878-7b6a3fb5ff56","Type":"ContainerStarted","Data":"ab07be6da2c3193d937e8c65312bd522aba99e6ec17de5832b29db2200ba2df9"} Jul 07 19:14:57.867304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:57.866953 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:14:57.903164 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:14:57.903093 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" podStartSLOduration=2.603992678 podStartE2EDuration="6.903073097s" totalImagesPullingTime="3.289924955s" totalInitContainerRuntime="1.009155464s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:57.902063672 +0000 UTC m=+343.252711893" watchObservedRunningTime="2026-07-07 19:14:57.903073097 +0000 UTC m=+343.253721337" Jul 07 19:15:00.006009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:00.005973 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:00.117269 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:00.117237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:03.044734 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:03.044693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:03.134539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:03.134502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:08.874435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:08.874352 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75847b84df-hccdw" Jul 07 19:15:11.161379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.161340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:11.296100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.296064 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:11.631401 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.631355 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:11.726068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.726029 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh"] Jul 07 19:15:11.728689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.728671 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.731477 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.731444 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 19:15:11.731657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.731539 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 19:15:11.731657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.731552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 19:15:11.731848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.731827 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-29rlv\"" Jul 07 19:15:11.731979 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.731853 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 19:15:11.744539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.744504 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh"] Jul 07 19:15:11.808929 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.808885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.809129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.808959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68q6n\" (UniqueName: \"kubernetes.io/projected/58177ee8-930d-43ed-8050-3ad7fbd97d26-kube-api-access-68q6n\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.809129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.809029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58177ee8-930d-43ed-8050-3ad7fbd97d26-tmp\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.909887 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.909796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58177ee8-930d-43ed-8050-3ad7fbd97d26-tmp\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.909887 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.909855 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.910168 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:11.909966 2568 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 07 19:15:11.910168 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.909998 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68q6n\" (UniqueName: \"kubernetes.io/projected/58177ee8-930d-43ed-8050-3ad7fbd97d26-kube-api-access-68q6n\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.910168 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:11.910032 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs podName:58177ee8-930d-43ed-8050-3ad7fbd97d26 nodeName:}" failed. No retries permitted until 2026-07-07 19:15:12.41001 +0000 UTC m=+357.760658215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs") pod "kube-auth-proxy-598d7cb498-4c7nh" (UID: "58177ee8-930d-43ed-8050-3ad7fbd97d26") : secret "kube-auth-proxy-tls" not found Jul 07 19:15:11.912429 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.912403 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58177ee8-930d-43ed-8050-3ad7fbd97d26-tmp\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:11.932174 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:11.932138 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68q6n\" (UniqueName: \"kubernetes.io/projected/58177ee8-930d-43ed-8050-3ad7fbd97d26-kube-api-access-68q6n\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:12.150048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.150011 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6qj6"] Jul 07 19:15:12.152782 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.152762 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.155150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.155117 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-plcz6\"" Jul 07 19:15:12.155766 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.155750 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 19:15:12.168498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.168463 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6qj6"] Jul 07 19:15:12.204998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.204958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:12.212032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.211991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrlm\" (UniqueName: \"kubernetes.io/projected/9b946389-9f25-40f3-ba29-b4da40029811-kube-api-access-rtrlm\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.212131 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.212074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.307187 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.307150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:12.313298 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.313268 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.313424 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.313374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrlm\" (UniqueName: \"kubernetes.io/projected/9b946389-9f25-40f3-ba29-b4da40029811-kube-api-access-rtrlm\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.313472 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:12.313422 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 19:15:12.313512 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:12.313498 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert podName:9b946389-9f25-40f3-ba29-b4da40029811 nodeName:}" failed. No retries permitted until 2026-07-07 19:15:12.813477959 +0000 UTC m=+358.164126176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert") pod "odh-model-controller-7998c67dc4-t6qj6" (UID: "9b946389-9f25-40f3-ba29-b4da40029811") : secret "odh-model-controller-webhook-cert" not found Jul 07 19:15:12.329639 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.329602 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrlm\" (UniqueName: \"kubernetes.io/projected/9b946389-9f25-40f3-ba29-b4da40029811-kube-api-access-rtrlm\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.414390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.414297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:12.416823 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.416798 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58177ee8-930d-43ed-8050-3ad7fbd97d26-tls-certs\") pod \"kube-auth-proxy-598d7cb498-4c7nh\" (UID: \"58177ee8-930d-43ed-8050-3ad7fbd97d26\") " pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:12.434731 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.434694 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:12.529707 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.529670 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:12.639506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.639474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" Jul 07 19:15:12.712700 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.712655 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:12.817833 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.817798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.820528 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.820506 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b946389-9f25-40f3-ba29-b4da40029811-cert\") pod \"odh-model-controller-7998c67dc4-t6qj6\" (UID: \"9b946389-9f25-40f3-ba29-b4da40029811\") " pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:12.821605 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:12.821578 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58177ee8_930d_43ed_8050_3ad7fbd97d26.slice/crio-ba0ac78eee0f58836588c489f3fe0481ff6445c4548c68b239de65102d2ef9c7 WatchSource:0}: Error finding container ba0ac78eee0f58836588c489f3fe0481ff6445c4548c68b239de65102d2ef9c7: Status 404 returned error can't find the container with id ba0ac78eee0f58836588c489f3fe0481ff6445c4548c68b239de65102d2ef9c7 Jul 07 19:15:12.821847 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.821828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh"] Jul 07 19:15:12.938070 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:12.938030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" event={"ID":"58177ee8-930d-43ed-8050-3ad7fbd97d26","Type":"ContainerStarted","Data":"ba0ac78eee0f58836588c489f3fe0481ff6445c4548c68b239de65102d2ef9c7"} Jul 07 19:15:13.064980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.064849 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:13.212573 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.212547 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t6qj6"] Jul 07 19:15:13.215077 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:13.215048 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b946389_9f25_40f3_ba29_b4da40029811.slice/crio-0136a3bd53caf93e29396f2071aae9ddafa3bc0951e766e7d03da6495bf21c14 WatchSource:0}: Error finding container 0136a3bd53caf93e29396f2071aae9ddafa3bc0951e766e7d03da6495bf21c14: Status 404 returned error can't find the container with id 0136a3bd53caf93e29396f2071aae9ddafa3bc0951e766e7d03da6495bf21c14 Jul 07 19:15:13.384220 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.384190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:13.411002 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.410968 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:13.413967 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.413942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.417152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.417125 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 19:15:13.417152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.417125 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v82sb\"" Jul 07 19:15:13.417830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.417807 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:15:13.417830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.417821 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 19:15:13.437996 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.437955 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:13.442179 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.442147 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 19:15:13.516435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.516394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:13.524884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.524850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.524897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.525019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.525050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525184 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.525143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknzq\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525218 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.525192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.525252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.525239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.626786 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.626732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.626797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.626845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.626902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.626954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.627007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknzq\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627305 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.627041 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.627765 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.627738 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.629953 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.629869 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.629953 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.629899 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.630164 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.629982 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.630479 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.630437 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.639277 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.639185 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.659335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.659301 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknzq\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq\") pod \"istiod-openshift-gateway-7574459656-x66kl\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.690225 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.690190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:13.724800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.724768 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:13.886376 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.886179 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:13.888536 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:13.888502 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515e268e_0b7d_4fc2_bcb1_73c6f3238220.slice/crio-1cd4e2061547017c60acd2514e1a4e066c5271700a30221d557f16a31894bfd1 WatchSource:0}: Error finding container 1cd4e2061547017c60acd2514e1a4e066c5271700a30221d557f16a31894bfd1: Status 404 returned error can't find the container with id 1cd4e2061547017c60acd2514e1a4e066c5271700a30221d557f16a31894bfd1 Jul 07 19:15:13.946976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.946912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" event={"ID":"9b946389-9f25-40f3-ba29-b4da40029811","Type":"ContainerStarted","Data":"0136a3bd53caf93e29396f2071aae9ddafa3bc0951e766e7d03da6495bf21c14"} Jul 07 19:15:13.948507 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:13.948476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" event={"ID":"515e268e-0b7d-4fc2-bcb1-73c6f3238220","Type":"ContainerStarted","Data":"1cd4e2061547017c60acd2514e1a4e066c5271700a30221d557f16a31894bfd1"} Jul 07 19:15:14.811312 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:14.811256 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:16.673800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:16.673760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:16.732572 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:16.732535 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:17.867373 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.867338 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:15:17.867773 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.867407 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:15:17.875145 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.875123 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 19:15:17.973493 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.973455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" event={"ID":"58177ee8-930d-43ed-8050-3ad7fbd97d26","Type":"ContainerStarted","Data":"cadc532326cd49ee33bc40b93cea613c681a94473f127c7629b5326b9794967a"} Jul 07 19:15:17.975660 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.975621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" event={"ID":"9b946389-9f25-40f3-ba29-b4da40029811","Type":"ContainerStarted","Data":"0ad696551e18d0eb872f2f817787ead3c5f3c09e02cef2178197ee2b04019314"} Jul 07 19:15:17.975819 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.975691 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:17.977282 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.977254 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" event={"ID":"515e268e-0b7d-4fc2-bcb1-73c6f3238220","Type":"ContainerStarted","Data":"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6"} Jul 07 19:15:17.977675 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.977648 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:17.991274 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:17.991211 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-598d7cb498-4c7nh" podStartSLOduration=1.941938849 podStartE2EDuration="6.991190516s" totalImagesPullingTime="5.049251667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:17.990784615 +0000 UTC m=+363.341432855" watchObservedRunningTime="2026-07-07 19:15:17.991190516 +0000 UTC m=+363.341838755" Jul 07 19:15:18.009656 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.009604 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" podStartSLOduration=1.363943095 podStartE2EDuration="6.009586553s" totalImagesPullingTime="4.645643458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:18.008303134 +0000 UTC m=+363.358951371" watchObservedRunningTime="2026-07-07 19:15:18.009586553 +0000 UTC m=+363.360234790" Jul 07 19:15:18.029394 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.029343 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" podStartSLOduration=1.053257821 podStartE2EDuration="5.029323861s" totalImagesPullingTime="3.97606604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:18.027338062 +0000 UTC m=+363.377986344" watchObservedRunningTime="2026-07-07 19:15:18.029323861 +0000 UTC m=+363.379972075" Jul 07 19:15:18.268949 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.268881 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.373892 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.373850 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.470427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.470393 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.713604 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.713571 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw"] Jul 07 19:15:18.716075 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.716057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.718330 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.718300 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nxznz\"" Jul 07 19:15:18.731166 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.731128 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw"] Jul 07 19:15:18.739293 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.739259 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.775933 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.775881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae484-227d-4297-a00c-303a4cce288b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.775933 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.775940 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.775961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvjg\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-kube-api-access-4gvjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae484-227d-4297-a00c-303a4cce288b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.776379 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.776247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.814451 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.814419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.876683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae484-227d-4297-a00c-303a4cce288b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.876683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvjg\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-kube-api-access-4gvjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.876947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae484-227d-4297-a00c-303a4cce288b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877289 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877350 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877419 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877553 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.877597 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.877555 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5ae484-227d-4297-a00c-303a4cce288b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.879203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.879180 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5ae484-227d-4297-a00c-303a4cce288b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.879678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.879653 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5ae484-227d-4297-a00c-303a4cce288b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.888232 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.888198 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvjg\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-kube-api-access-4gvjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.888374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.888259 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5ae484-227d-4297-a00c-303a4cce288b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qdrdw\" (UID: \"ad5ae484-227d-4297-a00c-303a4cce288b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:18.915455 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.915418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:18.984385 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:18.984311 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:19.026347 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:19.026303 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:19.197099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:19.197058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw"] Jul 07 19:15:19.208010 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:19.207977 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5ae484_227d_4297_a00c_303a4cce288b.slice/crio-b3b3287179fcc2133916aa0e6ade3f92a45101cdf23143d49c7573ae697a3097 WatchSource:0}: Error finding container b3b3287179fcc2133916aa0e6ade3f92a45101cdf23143d49c7573ae697a3097: Status 404 returned error can't find the container with id b3b3287179fcc2133916aa0e6ade3f92a45101cdf23143d49c7573ae697a3097 Jul 07 19:15:19.990506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:19.990456 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" event={"ID":"ad5ae484-227d-4297-a00c-303a4cce288b","Type":"ContainerStarted","Data":"b3b3287179fcc2133916aa0e6ade3f92a45101cdf23143d49c7573ae697a3097"} Jul 07 19:15:21.569372 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:21.569335 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:15:21.569674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:21.569409 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:15:21.569674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:21.569440 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:15:22.001274 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.001243 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" event={"ID":"ad5ae484-227d-4297-a00c-303a4cce288b","Type":"ContainerStarted","Data":"f678cb87b14dfec5da5d4158db7649e5a70ec0a7a8ce9966cb2e8f968ecb6e15"} Jul 07 19:15:22.027098 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.027057 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:22.032046 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.032017 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:22.034855 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.034795 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" podStartSLOduration=1.675754622 podStartE2EDuration="4.034768173s" totalImagesPullingTime="2.359013551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:22.034039567 +0000 UTC m=+367.384687826" watchObservedRunningTime="2026-07-07 19:15:22.034768173 +0000 UTC m=+367.385416422" Jul 07 19:15:22.425976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.425939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:22.504432 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.504387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:22.513901 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.513864 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2"] Jul 07 19:15:22.516192 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.516169 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p"] Jul 07 19:15:22.516341 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.516286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.518681 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.518657 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.519106 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.519077 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 19:15:22.519224 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.519165 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7p5j2\"" Jul 07 19:15:22.521045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.521025 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 19:15:22.521160 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.521046 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ksxm4\"" Jul 07 19:15:22.531835 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.531806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2"] Jul 07 19:15:22.533503 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.533478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p"] Jul 07 19:15:22.577771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.577733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:22.614073 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.614018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.614259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.614104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68vd\" (UniqueName: \"kubernetes.io/projected/b4be1f47-6af1-4444-8f7f-de7210d97298-kube-api-access-d68vd\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.614259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.614158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4be1f47-6af1-4444-8f7f-de7210d97298-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.614259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.614183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bl9c\" (UniqueName: \"kubernetes.io/projected/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-kube-api-access-5bl9c\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.641739 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.641697 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:22.714910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.714812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68vd\" (UniqueName: \"kubernetes.io/projected/b4be1f47-6af1-4444-8f7f-de7210d97298-kube-api-access-d68vd\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.714910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.714880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4be1f47-6af1-4444-8f7f-de7210d97298-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.714910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.714907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bl9c\" (UniqueName: \"kubernetes.io/projected/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-kube-api-access-5bl9c\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.715197 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.715001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.715197 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:22.715131 2568 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 19:15:22.715197 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:22.715197 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert podName:dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7 nodeName:}" failed. No retries permitted until 2026-07-07 19:15:23.215174652 +0000 UTC m=+368.565822869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert") pod "llmisvc-controller-manager-777c6bd7d5-kmv9p" (UID: "dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7") : secret "llmisvc-webhook-server-cert" not found Jul 07 19:15:22.717639 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.717608 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4be1f47-6af1-4444-8f7f-de7210d97298-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.725368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.725336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bl9c\" (UniqueName: \"kubernetes.io/projected/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-kube-api-access-5bl9c\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:22.725521 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.725338 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68vd\" (UniqueName: \"kubernetes.io/projected/b4be1f47-6af1-4444-8f7f-de7210d97298-kube-api-access-d68vd\") pod \"kserve-controller-manager-d6f4b65b8-gz2s2\" (UID: \"b4be1f47-6af1-4444-8f7f-de7210d97298\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.804792 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.804756 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:22.831809 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.831775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:22.985814 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:22.985786 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2"] Jul 07 19:15:22.991206 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:22.991165 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4be1f47_6af1_4444_8f7f_de7210d97298.slice/crio-b4188addac3b0cf29e6b396c1ca9c95755dfc0a261ba7768d424d0b9d376c89b WatchSource:0}: Error finding container b4188addac3b0cf29e6b396c1ca9c95755dfc0a261ba7768d424d0b9d376c89b: Status 404 returned error can't find the container with id b4188addac3b0cf29e6b396c1ca9c95755dfc0a261ba7768d424d0b9d376c89b Jul 07 19:15:23.009181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.009147 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" event={"ID":"b4be1f47-6af1-4444-8f7f-de7210d97298","Type":"ContainerStarted","Data":"b4188addac3b0cf29e6b396c1ca9c95755dfc0a261ba7768d424d0b9d376c89b"} Jul 07 19:15:23.009455 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.009435 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:23.010468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.010447 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qdrdw" Jul 07 19:15:23.219060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.219022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:23.221524 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.221492 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kmv9p\" (UID: \"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:23.439025 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.438982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:23.614530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.614498 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p"] Jul 07 19:15:23.616615 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:15:23.616583 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddbcdf5de_94d7_4c6f_bb0a_2a5510e7ade7.slice/crio-799dce450636047a4193b5d623fadc5255f6043fe50379108525ea955bd83c21 WatchSource:0}: Error finding container 799dce450636047a4193b5d623fadc5255f6043fe50379108525ea955bd83c21: Status 404 returned error can't find the container with id 799dce450636047a4193b5d623fadc5255f6043fe50379108525ea955bd83c21 Jul 07 19:15:23.731313 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.731203 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:23.829483 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.829442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:23.939530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:23.939494 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:24.016382 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:24.016302 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" event={"ID":"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7","Type":"ContainerStarted","Data":"799dce450636047a4193b5d623fadc5255f6043fe50379108525ea955bd83c21"} Jul 07 19:15:24.597221 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:24.597186 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:24.597469 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:24.597447 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerName="discovery" containerID="cri-o://86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6" gracePeriod=30 Jul 07 19:15:24.996148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:24.996119 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:25.024356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.024320 2568 generic.go:356] "Generic (PLEG): container finished" podID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerID="86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6" exitCode=0 Jul 07 19:15:25.024538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.024417 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" Jul 07 19:15:25.024538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.024453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" event={"ID":"515e268e-0b7d-4fc2-bcb1-73c6f3238220","Type":"ContainerDied","Data":"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6"} Jul 07 19:15:25.024538 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.024513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" event={"ID":"515e268e-0b7d-4fc2-bcb1-73c6f3238220","Type":"ContainerDied","Data":"1cd4e2061547017c60acd2514e1a4e066c5271700a30221d557f16a31894bfd1"} Jul 07 19:15:25.024676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.024540 2568 scope.go:117] "RemoveContainer" containerID="86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6" Jul 07 19:15:25.040525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040495 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040533 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pknzq\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040577 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040604 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040662 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040708 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.040780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.040751 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs\") pod \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\" (UID: \"515e268e-0b7d-4fc2-bcb1-73c6f3238220\") " Jul 07 19:15:25.042968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.042905 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:15:25.044630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.044592 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:15:25.044783 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.044611 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts" (OuterVolumeSpecName: "cacerts") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:15:25.045392 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.045362 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:15:25.045646 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.045626 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token" (OuterVolumeSpecName: "istio-token") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:15:25.045841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.045820 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq" (OuterVolumeSpecName: "kube-api-access-pknzq") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "kube-api-access-pknzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:15:25.046115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.046094 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs" (OuterVolumeSpecName: "local-certs") pod "515e268e-0b7d-4fc2-bcb1-73c6f3238220" (UID: "515e268e-0b7d-4fc2-bcb1-73c6f3238220"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:15:25.131321 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.131209 2568 scope.go:117] "RemoveContainer" containerID="86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6" Jul 07 19:15:25.131701 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:15:25.131670 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6\": container with ID starting with 86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6 not found: ID does not exist" containerID="86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6" Jul 07 19:15:25.131808 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.131715 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6"} err="failed to get container status \"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6\": rpc error: code = NotFound desc = could not find container \"86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6\": container with ID starting with 86353104a71d9f5c6027069b8888ae04ce24ddb9ba4c99e416c9b378518f0bc6 not found: ID does not exist" Jul 07 19:15:25.142307 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142273 2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/515e268e-0b7d-4fc2-bcb1-73c6f3238220-local-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142307 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142308 2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-kubeconfig\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142307 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142320 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pknzq\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-kube-api-access-pknzq\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142550 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142337 2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-cacerts\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142550 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142349 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-dns-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142550 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142361 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.142550 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.142371 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/515e268e-0b7d-4fc2-bcb1-73c6f3238220-istio-csr-ca-configmap\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:15:25.358034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.357999 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:25.360670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:25.360640 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x66kl"] Jul 07 19:15:27.038451 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.038338 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" event={"ID":"b4be1f47-6af1-4444-8f7f-de7210d97298","Type":"ContainerStarted","Data":"1e691b291283eb3ae5be70488d180fc6ad260d06af003bee8dd4ecaf11a03bed"} Jul 07 19:15:27.038888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.038489 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:15:27.040180 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.040153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" event={"ID":"dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7","Type":"ContainerStarted","Data":"0b269ca5be843d44ac3e8dc4d1edc94195e2647e98f6ed314b92519cf5ae9ff3"} Jul 07 19:15:27.040302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.040258 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:27.061863 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.061802 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" podStartSLOduration=1.310977522 podStartE2EDuration="5.061784158s" totalImagesPullingTime="3.750806636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:27.060431432 +0000 UTC m=+372.411079688" watchObservedRunningTime="2026-07-07 19:15:27.061784158 +0000 UTC m=+372.412432395" Jul 07 19:15:27.079661 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.079611 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" podStartSLOduration=1.9517705969999999 podStartE2EDuration="5.079592187s" totalImagesPullingTime="3.12782159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:15:27.078895727 +0000 UTC m=+372.429543982" watchObservedRunningTime="2026-07-07 19:15:27.079592187 +0000 UTC m=+372.430240419" Jul 07 19:15:27.178229 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.178193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:15:27.352347 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:27.352266 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" path="/var/lib/kubelet/pods/515e268e-0b7d-4fc2-bcb1-73c6f3238220/volumes" Jul 07 19:15:28.984814 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:28.984782 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-t6qj6" Jul 07 19:15:29.983456 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:29.983393 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-x66kl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 19:15:29.983645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:29.983487 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x66kl" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 19:15:58.047641 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:58.047602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kmv9p" Jul 07 19:15:58.050670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:15:58.050646 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz2s2" Jul 07 19:16:12.659938 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.659862 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:12.753531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.753497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:12.788505 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.788465 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:12.788895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.788878 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerName="discovery" Jul 07 19:16:12.788895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.788897 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerName="discovery" Jul 07 19:16:12.789078 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.789006 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="515e268e-0b7d-4fc2-bcb1-73c6f3238220" containerName="discovery" Jul 07 19:16:12.791171 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.791143 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:12.793619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.793589 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:16:12.793619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.793607 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:16:12.793831 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.793660 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-l826b\"" Jul 07 19:16:12.807750 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.807719 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:12.850249 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.850216 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:12.869970 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.869906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4pz\" (UniqueName: \"kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz\") pod \"kuadrant-operator-catalog-nvcdx\" (UID: \"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0\") " pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:12.970848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.970741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4pz\" (UniqueName: \"kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz\") pod \"kuadrant-operator-catalog-nvcdx\" (UID: \"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0\") " pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:12.980127 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:12.980093 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4pz\" (UniqueName: \"kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz\") pod \"kuadrant-operator-catalog-nvcdx\" (UID: \"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0\") " pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:13.101898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.101856 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:13.146841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.146805 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:13.248297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.248160 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:13.251104 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:13.251067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bfc1d9_0a8b_45eb_8089_0e5d6ba3f0c0.slice/crio-066c3cc0366dd1369cffdf192ec71a249262bfba820d68789b638e56f6794261 WatchSource:0}: Error finding container 066c3cc0366dd1369cffdf192ec71a249262bfba820d68789b638e56f6794261: Status 404 returned error can't find the container with id 066c3cc0366dd1369cffdf192ec71a249262bfba820d68789b638e56f6794261 Jul 07 19:16:13.269109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.269074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" event={"ID":"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0","Type":"ContainerStarted","Data":"066c3cc0366dd1369cffdf192ec71a249262bfba820d68789b638e56f6794261"} Jul 07 19:16:13.355453 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.355416 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lg245"] Jul 07 19:16:13.358700 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.358681 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:13.368082 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.368054 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lg245"] Jul 07 19:16:13.474999 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.474962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfdl\" (UniqueName: \"kubernetes.io/projected/0a419f62-2cc2-42a1-8e64-f00bbee05709-kube-api-access-gxfdl\") pod \"kuadrant-operator-catalog-lg245\" (UID: \"0a419f62-2cc2-42a1-8e64-f00bbee05709\") " pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:13.576399 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.576299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfdl\" (UniqueName: \"kubernetes.io/projected/0a419f62-2cc2-42a1-8e64-f00bbee05709-kube-api-access-gxfdl\") pod \"kuadrant-operator-catalog-lg245\" (UID: \"0a419f62-2cc2-42a1-8e64-f00bbee05709\") " pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:13.586069 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.586030 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfdl\" (UniqueName: \"kubernetes.io/projected/0a419f62-2cc2-42a1-8e64-f00bbee05709-kube-api-access-gxfdl\") pod \"kuadrant-operator-catalog-lg245\" (UID: \"0a419f62-2cc2-42a1-8e64-f00bbee05709\") " pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:13.669510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.669468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:13.833305 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:13.833280 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lg245"] Jul 07 19:16:13.861454 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:13.861423 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a419f62_2cc2_42a1_8e64_f00bbee05709.slice/crio-5be0a80d4e11d49e217b3d43fb1fcd8cfc61ccc10120c1a740b260b7bb186cfe WatchSource:0}: Error finding container 5be0a80d4e11d49e217b3d43fb1fcd8cfc61ccc10120c1a740b260b7bb186cfe: Status 404 returned error can't find the container with id 5be0a80d4e11d49e217b3d43fb1fcd8cfc61ccc10120c1a740b260b7bb186cfe Jul 07 19:16:14.278495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:14.278453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lg245" event={"ID":"0a419f62-2cc2-42a1-8e64-f00bbee05709","Type":"ContainerStarted","Data":"5be0a80d4e11d49e217b3d43fb1fcd8cfc61ccc10120c1a740b260b7bb186cfe"} Jul 07 19:16:15.286106 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.286028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lg245" event={"ID":"0a419f62-2cc2-42a1-8e64-f00bbee05709","Type":"ContainerStarted","Data":"43ffe785ac74153452f14ac6841d77390da46a22286a992082875d541462af5c"} Jul 07 19:16:15.287798 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.287768 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" event={"ID":"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0","Type":"ContainerStarted","Data":"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16"} Jul 07 19:16:15.287960 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.287834 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" podUID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" containerName="registry-server" containerID="cri-o://30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16" gracePeriod=2 Jul 07 19:16:15.305216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.305166 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lg245" podStartSLOduration=1.05921195 podStartE2EDuration="2.30514757s" totalImagesPullingTime="1.24593562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:15.304597422 +0000 UTC m=+420.655245682" watchObservedRunningTime="2026-07-07 19:16:15.30514757 +0000 UTC m=+420.655795806" Jul 07 19:16:15.322575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.322518 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" podStartSLOduration=1.46622706 podStartE2EDuration="3.322500612s" totalImagesPullingTime="1.856273552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:15.321784002 +0000 UTC m=+420.672432260" watchObservedRunningTime="2026-07-07 19:16:15.322500612 +0000 UTC m=+420.673148850" Jul 07 19:16:15.536697 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.536669 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:15.597816 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.597781 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb4pz\" (UniqueName: \"kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz\") pod \"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0\" (UID: \"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0\") " Jul 07 19:16:15.600072 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.600041 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz" (OuterVolumeSpecName: "kube-api-access-cb4pz") pod "f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" (UID: "f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0"). InnerVolumeSpecName "kube-api-access-cb4pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:15.698623 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:15.698504 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4pz\" (UniqueName: \"kubernetes.io/projected/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0-kube-api-access-cb4pz\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:16.293486 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.293446 2568 generic.go:356] "Generic (PLEG): container finished" podID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" containerID="30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16" exitCode=0 Jul 07 19:16:16.293900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.293510 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" Jul 07 19:16:16.293900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.293535 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" event={"ID":"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0","Type":"ContainerDied","Data":"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16"} Jul 07 19:16:16.293900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.293627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nvcdx" event={"ID":"f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0","Type":"ContainerDied","Data":"066c3cc0366dd1369cffdf192ec71a249262bfba820d68789b638e56f6794261"} Jul 07 19:16:16.293900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.293648 2568 scope.go:117] "RemoveContainer" containerID="30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16" Jul 07 19:16:16.303146 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.303125 2568 scope.go:117] "RemoveContainer" containerID="30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16" Jul 07 19:16:16.303518 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:16:16.303493 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16\": container with ID starting with 30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16 not found: ID does not exist" containerID="30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16" Jul 07 19:16:16.303561 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.303529 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16"} err="failed to get container status \"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16\": rpc error: code = NotFound desc = could not find container \"30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16\": container with ID starting with 30a5fcbc6f1b447b319eca24ccab0f3bff9c7f6926ab8dd269707f26d14e4d16 not found: ID does not exist" Jul 07 19:16:16.318607 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.318572 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:16.323638 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:16.323609 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nvcdx"] Jul 07 19:16:17.352163 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:17.352132 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" path="/var/lib/kubelet/pods/f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0/volumes" Jul 07 19:16:23.670580 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:23.670538 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:23.671034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:23.670686 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:23.693897 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:23.693862 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:24.357749 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:24.357719 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lg245" Jul 07 19:16:28.755902 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.755859 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4"] Jul 07 19:16:28.756456 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.756435 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" containerName="registry-server" Jul 07 19:16:28.756551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.756460 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" containerName="registry-server" Jul 07 19:16:28.756606 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.756587 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5bfc1d9-0a8b-45eb-8089-0e5d6ba3f0c0" containerName="registry-server" Jul 07 19:16:28.759842 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.759818 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:28.762443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.762421 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lcbmm\"" Jul 07 19:16:28.772129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.772095 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4"] Jul 07 19:16:28.923625 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.923580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:28.923824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.923638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcsz\" (UniqueName: \"kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:28.923824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:28.923767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.024941 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.024811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.024941 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.024863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcsz\" (UniqueName: \"kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.024941 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.024946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.025306 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.025287 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.025382 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.025310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.035827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.035790 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcsz\" (UniqueName: \"kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.069522 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.069483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:29.217103 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.217069 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4"] Jul 07 19:16:29.218952 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:29.218910 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3daa69_2f5b_497e_8e4d_f1c2ec6db037.slice/crio-dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452 WatchSource:0}: Error finding container dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452: Status 404 returned error can't find the container with id dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452 Jul 07 19:16:29.358718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.358682 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6"] Jul 07 19:16:29.360255 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.360226 2568 generic.go:356] "Generic (PLEG): container finished" podID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerID="6cf07c501d58642ff4154b30335a3511ea7e12c2faf42578407879286cbff359" exitCode=0 Jul 07 19:16:29.361584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.361561 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" event={"ID":"da3daa69-2f5b-497e-8e4d-f1c2ec6db037","Type":"ContainerDied","Data":"6cf07c501d58642ff4154b30335a3511ea7e12c2faf42578407879286cbff359"} Jul 07 19:16:29.361690 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.361600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" event={"ID":"da3daa69-2f5b-497e-8e4d-f1c2ec6db037","Type":"ContainerStarted","Data":"dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452"} Jul 07 19:16:29.361690 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.361685 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.374061 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.374028 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6"] Jul 07 19:16:29.529506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.529469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.529506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.529510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.529750 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.529599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qcb\" (UniqueName: \"kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.630118 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.630085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qcb\" (UniqueName: \"kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.630293 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.630164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.630293 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.630193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.630563 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.630546 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.630641 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.630618 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.640878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.640841 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qcb\" (UniqueName: \"kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.672173 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.672143 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:29.754399 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.754364 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk"] Jul 07 19:16:29.757431 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.757406 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.769430 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.769353 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk"] Jul 07 19:16:29.830900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.830875 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6"] Jul 07 19:16:29.832768 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.832738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.832930 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.832875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.833006 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.832975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf6z\" (UniqueName: \"kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.833006 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:29.832989 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cba973_55fe_46c2_982d_96b2c9630179.slice/crio-ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416 WatchSource:0}: Error finding container ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416: Status 404 returned error can't find the container with id ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416 Jul 07 19:16:29.934271 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.934231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf6z\" (UniqueName: \"kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.934611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.934301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.934611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.934413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.934794 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.934772 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.934852 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.934783 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:29.944037 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:29.944007 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf6z\" (UniqueName: \"kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:30.071756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.071717 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:30.169436 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.169397 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb"] Jul 07 19:16:30.173467 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.173444 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.186249 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.186216 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb"] Jul 07 19:16:30.220406 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.220369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk"] Jul 07 19:16:30.223848 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:30.223810 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545295a5_64a6_49ca_8134_565684dcb9e6.slice/crio-5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9 WatchSource:0}: Error finding container 5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9: Status 404 returned error can't find the container with id 5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9 Jul 07 19:16:30.338675 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.338641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.338857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.338694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gd4\" (UniqueName: \"kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.338857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.338807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.368111 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.368077 2568 generic.go:356] "Generic (PLEG): container finished" podID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerID="79737c46a8d5761cb78ca81154a2dde8b7b454ebb9f40ca38795b378ea2053c0" exitCode=0 Jul 07 19:16:30.368295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.368155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" event={"ID":"da3daa69-2f5b-497e-8e4d-f1c2ec6db037","Type":"ContainerDied","Data":"79737c46a8d5761cb78ca81154a2dde8b7b454ebb9f40ca38795b378ea2053c0"} Jul 07 19:16:30.369781 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.369753 2568 generic.go:356] "Generic (PLEG): container finished" podID="545295a5-64a6-49ca-8134-565684dcb9e6" containerID="0a4aaccf8bf0414987d936adf3385844a988c70c41b0cfbf91020739f305c70a" exitCode=0 Jul 07 19:16:30.369862 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.369836 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" event={"ID":"545295a5-64a6-49ca-8134-565684dcb9e6","Type":"ContainerDied","Data":"0a4aaccf8bf0414987d936adf3385844a988c70c41b0cfbf91020739f305c70a"} Jul 07 19:16:30.369957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.369894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" event={"ID":"545295a5-64a6-49ca-8134-565684dcb9e6","Type":"ContainerStarted","Data":"5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9"} Jul 07 19:16:30.371394 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.371370 2568 generic.go:356] "Generic (PLEG): container finished" podID="00cba973-55fe-46c2-982d-96b2c9630179" containerID="f36cd3d675d667ff2eb6356ef00fbbfb67bbd11b91f2952db37ab9147996659b" exitCode=0 Jul 07 19:16:30.371494 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.371458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" event={"ID":"00cba973-55fe-46c2-982d-96b2c9630179","Type":"ContainerDied","Data":"f36cd3d675d667ff2eb6356ef00fbbfb67bbd11b91f2952db37ab9147996659b"} Jul 07 19:16:30.371557 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.371509 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" event={"ID":"00cba973-55fe-46c2-982d-96b2c9630179","Type":"ContainerStarted","Data":"ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416"} Jul 07 19:16:30.440143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.440102 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.440349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.440161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gd4\" (UniqueName: \"kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.440349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.440264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.440601 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.440570 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.440745 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.440611 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.454395 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.454352 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gd4\" (UniqueName: \"kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.486711 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.486676 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:30.630510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:30.630477 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb"] Jul 07 19:16:30.631883 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:30.631849 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc51cc1_7e44_4aff_a809_0cf8daab9e35.slice/crio-b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462 WatchSource:0}: Error finding container b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462: Status 404 returned error can't find the container with id b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462 Jul 07 19:16:31.378617 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.378584 2568 generic.go:356] "Generic (PLEG): container finished" podID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerID="5af76c8b923e69ee0ac4cb9cb3a45c39fc6fc6cdf3f1f7f24454ed2e645cf3b0" exitCode=0 Jul 07 19:16:31.380792 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.379065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" event={"ID":"da3daa69-2f5b-497e-8e4d-f1c2ec6db037","Type":"ContainerDied","Data":"5af76c8b923e69ee0ac4cb9cb3a45c39fc6fc6cdf3f1f7f24454ed2e645cf3b0"} Jul 07 19:16:31.387095 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.387062 2568 generic.go:356] "Generic (PLEG): container finished" podID="545295a5-64a6-49ca-8134-565684dcb9e6" containerID="587ca289a0ffbeb2268cbe904ac45ddb76f385834bdcb3f16a8df004c5ed9d0c" exitCode=0 Jul 07 19:16:31.387318 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.387155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" event={"ID":"545295a5-64a6-49ca-8134-565684dcb9e6","Type":"ContainerDied","Data":"587ca289a0ffbeb2268cbe904ac45ddb76f385834bdcb3f16a8df004c5ed9d0c"} Jul 07 19:16:31.389515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.389453 2568 generic.go:356] "Generic (PLEG): container finished" podID="00cba973-55fe-46c2-982d-96b2c9630179" containerID="f37aed9fcf1ee0357ff7d8f7aaf7f1236d1cafe8529238fc9d4e8ccd2ccc49a2" exitCode=0 Jul 07 19:16:31.389515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.389492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" event={"ID":"00cba973-55fe-46c2-982d-96b2c9630179","Type":"ContainerDied","Data":"f37aed9fcf1ee0357ff7d8f7aaf7f1236d1cafe8529238fc9d4e8ccd2ccc49a2"} Jul 07 19:16:31.391333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.391309 2568 generic.go:356] "Generic (PLEG): container finished" podID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerID="d6c4a814c8d80f5807e4f7a678ed8c8efe3ca3d226838b7c68f23fc4b4bb5436" exitCode=0 Jul 07 19:16:31.391418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.391350 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" event={"ID":"2cc51cc1-7e44-4aff-a809-0cf8daab9e35","Type":"ContainerDied","Data":"d6c4a814c8d80f5807e4f7a678ed8c8efe3ca3d226838b7c68f23fc4b4bb5436"} Jul 07 19:16:31.391418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:31.391387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" event={"ID":"2cc51cc1-7e44-4aff-a809-0cf8daab9e35","Type":"ContainerStarted","Data":"b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462"} Jul 07 19:16:32.398890 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.398852 2568 generic.go:356] "Generic (PLEG): container finished" podID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerID="41d709f517fc04db028a62ebb7845c249faa71dc56db85e7752cc20e2cbfc9b4" exitCode=0 Jul 07 19:16:32.399396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.398955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" event={"ID":"2cc51cc1-7e44-4aff-a809-0cf8daab9e35","Type":"ContainerDied","Data":"41d709f517fc04db028a62ebb7845c249faa71dc56db85e7752cc20e2cbfc9b4"} Jul 07 19:16:32.401395 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.401366 2568 generic.go:356] "Generic (PLEG): container finished" podID="545295a5-64a6-49ca-8134-565684dcb9e6" containerID="4aa8c670ce03d9e6836215818e371beee1598b11b635320b4b2c8bf76ab1d0b7" exitCode=0 Jul 07 19:16:32.401498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.401446 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" event={"ID":"545295a5-64a6-49ca-8134-565684dcb9e6","Type":"ContainerDied","Data":"4aa8c670ce03d9e6836215818e371beee1598b11b635320b4b2c8bf76ab1d0b7"} Jul 07 19:16:32.406849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.406821 2568 generic.go:356] "Generic (PLEG): container finished" podID="00cba973-55fe-46c2-982d-96b2c9630179" containerID="46ba6a3f8d8fe06623d75a3487670e7e30d6a93f5944aceed351556f0772e3a3" exitCode=0 Jul 07 19:16:32.407007 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.406941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" event={"ID":"00cba973-55fe-46c2-982d-96b2c9630179","Type":"ContainerDied","Data":"46ba6a3f8d8fe06623d75a3487670e7e30d6a93f5944aceed351556f0772e3a3"} Jul 07 19:16:32.555541 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.555515 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:32.663444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.663349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle\") pod \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " Jul 07 19:16:32.663444 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.663417 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxcsz\" (UniqueName: \"kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz\") pod \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " Jul 07 19:16:32.663644 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.663494 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util\") pod \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\" (UID: \"da3daa69-2f5b-497e-8e4d-f1c2ec6db037\") " Jul 07 19:16:32.663889 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.663857 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle" (OuterVolumeSpecName: "bundle") pod "da3daa69-2f5b-497e-8e4d-f1c2ec6db037" (UID: "da3daa69-2f5b-497e-8e4d-f1c2ec6db037"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:32.665668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.665635 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz" (OuterVolumeSpecName: "kube-api-access-dxcsz") pod "da3daa69-2f5b-497e-8e4d-f1c2ec6db037" (UID: "da3daa69-2f5b-497e-8e4d-f1c2ec6db037"). InnerVolumeSpecName "kube-api-access-dxcsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:32.669384 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.669353 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util" (OuterVolumeSpecName: "util") pod "da3daa69-2f5b-497e-8e4d-f1c2ec6db037" (UID: "da3daa69-2f5b-497e-8e4d-f1c2ec6db037"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:32.764295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.764242 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:32.764295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.764284 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:32.764295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:32.764297 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxcsz\" (UniqueName: \"kubernetes.io/projected/da3daa69-2f5b-497e-8e4d-f1c2ec6db037-kube-api-access-dxcsz\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.414164 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.414131 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" Jul 07 19:16:33.414164 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.414148 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4" event={"ID":"da3daa69-2f5b-497e-8e4d-f1c2ec6db037","Type":"ContainerDied","Data":"dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452"} Jul 07 19:16:33.414663 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.414208 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfdb6504fc2134b552bd3a96db48876ea146a3773045f2afcfc94917566a7452" Jul 07 19:16:33.416656 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.416629 2568 generic.go:356] "Generic (PLEG): container finished" podID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerID="443e08e19587411e3906d7e42153b21845184e1ab4610878720cd1b761daa721" exitCode=0 Jul 07 19:16:33.416850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.416683 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" event={"ID":"2cc51cc1-7e44-4aff-a809-0cf8daab9e35","Type":"ContainerDied","Data":"443e08e19587411e3906d7e42153b21845184e1ab4610878720cd1b761daa721"} Jul 07 19:16:33.570948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.570909 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:33.590556 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.590527 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:33.670364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.670260 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util\") pod \"545295a5-64a6-49ca-8134-565684dcb9e6\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " Jul 07 19:16:33.670364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.670310 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle\") pod \"545295a5-64a6-49ca-8134-565684dcb9e6\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " Jul 07 19:16:33.670594 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.670408 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnf6z\" (UniqueName: \"kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z\") pod \"545295a5-64a6-49ca-8134-565684dcb9e6\" (UID: \"545295a5-64a6-49ca-8134-565684dcb9e6\") " Jul 07 19:16:33.671003 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.670963 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle" (OuterVolumeSpecName: "bundle") pod "545295a5-64a6-49ca-8134-565684dcb9e6" (UID: "545295a5-64a6-49ca-8134-565684dcb9e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:33.672644 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.672623 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z" (OuterVolumeSpecName: "kube-api-access-fnf6z") pod "545295a5-64a6-49ca-8134-565684dcb9e6" (UID: "545295a5-64a6-49ca-8134-565684dcb9e6"). InnerVolumeSpecName "kube-api-access-fnf6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:33.675953 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.675895 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util" (OuterVolumeSpecName: "util") pod "545295a5-64a6-49ca-8134-565684dcb9e6" (UID: "545295a5-64a6-49ca-8134-565684dcb9e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:33.771027 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.770988 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle\") pod \"00cba973-55fe-46c2-982d-96b2c9630179\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " Jul 07 19:16:33.771203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771043 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util\") pod \"00cba973-55fe-46c2-982d-96b2c9630179\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " Jul 07 19:16:33.771203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771100 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5qcb\" (UniqueName: \"kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb\") pod \"00cba973-55fe-46c2-982d-96b2c9630179\" (UID: \"00cba973-55fe-46c2-982d-96b2c9630179\") " Jul 07 19:16:33.771365 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771349 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnf6z\" (UniqueName: \"kubernetes.io/projected/545295a5-64a6-49ca-8134-565684dcb9e6-kube-api-access-fnf6z\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.771420 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771371 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.771420 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771386 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/545295a5-64a6-49ca-8134-565684dcb9e6-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.771540 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.771513 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle" (OuterVolumeSpecName: "bundle") pod "00cba973-55fe-46c2-982d-96b2c9630179" (UID: "00cba973-55fe-46c2-982d-96b2c9630179"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:33.773194 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.773164 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb" (OuterVolumeSpecName: "kube-api-access-h5qcb") pod "00cba973-55fe-46c2-982d-96b2c9630179" (UID: "00cba973-55fe-46c2-982d-96b2c9630179"). InnerVolumeSpecName "kube-api-access-h5qcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:33.776854 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.776809 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util" (OuterVolumeSpecName: "util") pod "00cba973-55fe-46c2-982d-96b2c9630179" (UID: "00cba973-55fe-46c2-982d-96b2c9630179"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:33.872850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.872814 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.872850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.872844 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00cba973-55fe-46c2-982d-96b2c9630179-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:33.872850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:33.872854 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qcb\" (UniqueName: \"kubernetes.io/projected/00cba973-55fe-46c2-982d-96b2c9630179-kube-api-access-h5qcb\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:34.017259 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.017174 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:34.424393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.424353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" event={"ID":"545295a5-64a6-49ca-8134-565684dcb9e6","Type":"ContainerDied","Data":"5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9"} Jul 07 19:16:34.424813 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.424405 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5df09d488a74d192702c8a900d6334012730ab4db0283b097dd59fda5626a9" Jul 07 19:16:34.424813 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.424365 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk" Jul 07 19:16:34.434166 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.434134 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" Jul 07 19:16:34.434330 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.434131 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6" event={"ID":"00cba973-55fe-46c2-982d-96b2c9630179","Type":"ContainerDied","Data":"ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416"} Jul 07 19:16:34.434330 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.434298 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae0895861fa49c13e676db91c3529b61234af2f44e494f9b43d8c7d8a1553416" Jul 07 19:16:34.574865 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.574839 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:34.679045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.678951 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle\") pod \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " Jul 07 19:16:34.679045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.679026 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5gd4\" (UniqueName: \"kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4\") pod \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " Jul 07 19:16:34.679045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.679046 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util\") pod \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\" (UID: \"2cc51cc1-7e44-4aff-a809-0cf8daab9e35\") " Jul 07 19:16:34.679715 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.679685 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle" (OuterVolumeSpecName: "bundle") pod "2cc51cc1-7e44-4aff-a809-0cf8daab9e35" (UID: "2cc51cc1-7e44-4aff-a809-0cf8daab9e35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:34.681189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.681160 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4" (OuterVolumeSpecName: "kube-api-access-c5gd4") pod "2cc51cc1-7e44-4aff-a809-0cf8daab9e35" (UID: "2cc51cc1-7e44-4aff-a809-0cf8daab9e35"). InnerVolumeSpecName "kube-api-access-c5gd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:16:34.684928 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.684879 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util" (OuterVolumeSpecName: "util") pod "2cc51cc1-7e44-4aff-a809-0cf8daab9e35" (UID: "2cc51cc1-7e44-4aff-a809-0cf8daab9e35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:16:34.780348 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.780301 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gd4\" (UniqueName: \"kubernetes.io/projected/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-kube-api-access-c5gd4\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:34.780348 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.780339 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-util\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:34.780348 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:34.780350 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cc51cc1-7e44-4aff-a809-0cf8daab9e35-bundle\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:16:35.441476 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:35.441448 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" Jul 07 19:16:35.441476 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:35.441465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb" event={"ID":"2cc51cc1-7e44-4aff-a809-0cf8daab9e35","Type":"ContainerDied","Data":"b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462"} Jul 07 19:16:35.441892 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:35.441505 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f4a7bbe676ed4ab3191522aac5fc0e3112dbfb737635af6570bd5d5d50d462" Jul 07 19:16:41.895531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:41.895444 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:45.272551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:45.272514 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:46.560613 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560577 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck"] Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560957 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="pull" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560969 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="pull" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560978 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560985 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.560996 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561002 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561009 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561015 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561026 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="pull" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561032 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="pull" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561037 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="extract" Jul 07 19:16:46.561043 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561041 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="extract" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561052 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561057 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561062 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="pull" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561066 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="pull" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561073 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561078 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561084 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561089 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561095 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="pull" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561100 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="pull" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561105 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561109 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="util" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561164 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cba973-55fe-46c2-982d-96b2c9630179" containerName="extract" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561170 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3daa69-2f5b-497e-8e4d-f1c2ec6db037" containerName="extract" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561177 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc51cc1-7e44-4aff-a809-0cf8daab9e35" containerName="extract" Jul 07 19:16:46.561412 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.561187 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="545295a5-64a6-49ca-8134-565684dcb9e6" containerName="extract" Jul 07 19:16:46.564445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.564425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.567128 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.567098 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q5b5p\"" Jul 07 19:16:46.579495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.579460 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck"] Jul 07 19:16:46.595195 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.595140 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgv5\" (UniqueName: \"kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.595408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.595291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.696120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.696081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.696321 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.696163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgv5\" (UniqueName: \"kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.696508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.696486 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.712216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.712181 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgv5\" (UniqueName: \"kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:46.828264 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.828175 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:46.875276 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:46.875235 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:47.025791 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:47.025756 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck"] Jul 07 19:16:47.027197 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:47.027170 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa48c651_7eed_4c94_92f6_8ba7e2a3e03e.slice/crio-21ff2b4a7c11039cfc7ba84f5c966cdbee4f72322b0dd3786f4be101b70023cd WatchSource:0}: Error finding container 21ff2b4a7c11039cfc7ba84f5c966cdbee4f72322b0dd3786f4be101b70023cd: Status 404 returned error can't find the container with id 21ff2b4a7c11039cfc7ba84f5c966cdbee4f72322b0dd3786f4be101b70023cd Jul 07 19:16:47.243356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:47.243315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:47.513789 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:47.513707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" event={"ID":"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e","Type":"ContainerStarted","Data":"21ff2b4a7c11039cfc7ba84f5c966cdbee4f72322b0dd3786f4be101b70023cd"} Jul 07 19:16:48.266560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:48.266514 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:49.226299 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.226258 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-57vn5"] Jul 07 19:16:49.233130 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.233100 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:49.235947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.235900 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2phb4\"" Jul 07 19:16:49.273680 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.273643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-57vn5"] Jul 07 19:16:49.322767 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.322729 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sbq\" (UniqueName: \"kubernetes.io/projected/a88e8519-3d49-40bc-bfb3-bd6d63f02df0-kube-api-access-45sbq\") pod \"authorino-operator-75cc8d5cb7-57vn5\" (UID: \"a88e8519-3d49-40bc-bfb3-bd6d63f02df0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:49.423629 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.423593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sbq\" (UniqueName: \"kubernetes.io/projected/a88e8519-3d49-40bc-bfb3-bd6d63f02df0-kube-api-access-45sbq\") pod \"authorino-operator-75cc8d5cb7-57vn5\" (UID: \"a88e8519-3d49-40bc-bfb3-bd6d63f02df0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:49.435886 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.435847 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sbq\" (UniqueName: \"kubernetes.io/projected/a88e8519-3d49-40bc-bfb3-bd6d63f02df0-kube-api-access-45sbq\") pod \"authorino-operator-75cc8d5cb7-57vn5\" (UID: \"a88e8519-3d49-40bc-bfb3-bd6d63f02df0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:49.545794 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.545697 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:49.706738 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.706707 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-57vn5"] Jul 07 19:16:49.708063 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:49.708021 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88e8519_3d49_40bc_bfb3_bd6d63f02df0.slice/crio-13f096b3c618aaf9055c3db815a7f8df330413eb46eb8b38059e33d0e273c92d WatchSource:0}: Error finding container 13f096b3c618aaf9055c3db815a7f8df330413eb46eb8b38059e33d0e273c92d: Status 404 returned error can't find the container with id 13f096b3c618aaf9055c3db815a7f8df330413eb46eb8b38059e33d0e273c92d Jul 07 19:16:49.824300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.824140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:49.922809 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:49.922768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:50.535007 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:50.534964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" event={"ID":"a88e8519-3d49-40bc-bfb3-bd6d63f02df0","Type":"ContainerStarted","Data":"13f096b3c618aaf9055c3db815a7f8df330413eb46eb8b38059e33d0e273c92d"} Jul 07 19:16:51.433341 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.433289 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t"] Jul 07 19:16:51.437335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.437296 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:51.440244 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.440217 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 19:16:51.440410 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.440217 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sclmp\"" Jul 07 19:16:51.450189 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.450154 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t"] Jul 07 19:16:51.543798 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.543751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ptb\" (UniqueName: \"kubernetes.io/projected/c4e19632-bc87-47d6-9c71-793768ad895d-kube-api-access-z6ptb\") pod \"dns-operator-controller-manager-864dd78c4c-sjr2t\" (UID: \"c4e19632-bc87-47d6-9c71-793768ad895d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:51.645775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.645732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ptb\" (UniqueName: \"kubernetes.io/projected/c4e19632-bc87-47d6-9c71-793768ad895d-kube-api-access-z6ptb\") pod \"dns-operator-controller-manager-864dd78c4c-sjr2t\" (UID: \"c4e19632-bc87-47d6-9c71-793768ad895d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:51.661091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.661059 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ptb\" (UniqueName: \"kubernetes.io/projected/c4e19632-bc87-47d6-9c71-793768ad895d-kube-api-access-z6ptb\") pod \"dns-operator-controller-manager-864dd78c4c-sjr2t\" (UID: \"c4e19632-bc87-47d6-9c71-793768ad895d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:51.752570 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:51.752479 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:52.031244 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:52.031159 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:53.309430 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.309400 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t"] Jul 07 19:16:53.311408 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:53.311371 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e19632_bc87_47d6_9c71_793768ad895d.slice/crio-eb2214a933f30cf0701cf8d8afee83a631d4771fc4379f4c75a02d37fe95e0c5 WatchSource:0}: Error finding container eb2214a933f30cf0701cf8d8afee83a631d4771fc4379f4c75a02d37fe95e0c5: Status 404 returned error can't find the container with id eb2214a933f30cf0701cf8d8afee83a631d4771fc4379f4c75a02d37fe95e0c5 Jul 07 19:16:53.528707 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.528603 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb"] Jul 07 19:16:53.532623 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.532601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:53.534840 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.534817 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kfp66\"" Jul 07 19:16:53.544032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.544001 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb"] Jul 07 19:16:53.556239 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.556200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" event={"ID":"a88e8519-3d49-40bc-bfb3-bd6d63f02df0","Type":"ContainerStarted","Data":"1476f682aeaf60cd7b06faab0009d3821a5668460cc2b5cc1b0027e730f98a87"} Jul 07 19:16:53.556459 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.556263 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:16:53.557836 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.557799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" event={"ID":"c4e19632-bc87-47d6-9c71-793768ad895d","Type":"ContainerStarted","Data":"eb2214a933f30cf0701cf8d8afee83a631d4771fc4379f4c75a02d37fe95e0c5"} Jul 07 19:16:53.559542 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.559519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" event={"ID":"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e","Type":"ContainerStarted","Data":"48750cdeb784813bd7817c62165e2b9aa696ba4e26c6a431596180c7265d644c"} Jul 07 19:16:53.559685 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.559667 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:16:53.580344 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.580274 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" podStartSLOduration=1.13472679 podStartE2EDuration="4.580255378s" totalImagesPullingTime="3.445528588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:53.578474852 +0000 UTC m=+458.929123099" watchObservedRunningTime="2026-07-07 19:16:53.580255378 +0000 UTC m=+458.930903615" Jul 07 19:16:53.600996 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.600933 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" podStartSLOduration=1.461475176 podStartE2EDuration="7.600897623s" totalImagesPullingTime="6.139422447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:53.599871086 +0000 UTC m=+458.950519344" watchObservedRunningTime="2026-07-07 19:16:53.600897623 +0000 UTC m=+458.951545859" Jul 07 19:16:53.665050 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.665002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zjm\" (UniqueName: \"kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm\") pod \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" (UID: \"286cf3ec-1384-41a9-905e-bbe14b930996\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:53.766268 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.766229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zjm\" (UniqueName: \"kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm\") pod \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" (UID: \"286cf3ec-1384-41a9-905e-bbe14b930996\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:53.777281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.777244 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zjm\" (UniqueName: \"kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm\") pod \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" (UID: \"286cf3ec-1384-41a9-905e-bbe14b930996\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:53.845609 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.845530 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:53.880537 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:53.880496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:54.230963 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:54.230668 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb"] Jul 07 19:16:54.242331 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:16:54.242294 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286cf3ec_1384_41a9_905e_bbe14b930996.slice/crio-5fed95f647a9d61c683f182f5adb70908aaab75357a93c98356e9160a0e013f9 WatchSource:0}: Error finding container 5fed95f647a9d61c683f182f5adb70908aaab75357a93c98356e9160a0e013f9: Status 404 returned error can't find the container with id 5fed95f647a9d61c683f182f5adb70908aaab75357a93c98356e9160a0e013f9 Jul 07 19:16:54.566586 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:54.566539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" event={"ID":"286cf3ec-1384-41a9-905e-bbe14b930996","Type":"ContainerStarted","Data":"5fed95f647a9d61c683f182f5adb70908aaab75357a93c98356e9160a0e013f9"} Jul 07 19:16:55.575524 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:55.575418 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" event={"ID":"c4e19632-bc87-47d6-9c71-793768ad895d","Type":"ContainerStarted","Data":"ca863c9c91f838c0c79b3e41aa68a2919096d17949677d28d6236e7a7883b8da"} Jul 07 19:16:55.576012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:55.575532 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:16:55.600996 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:55.600909 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" podStartSLOduration=2.577440901 podStartE2EDuration="4.600886835s" totalImagesPullingTime="2.023445934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:55.599639837 +0000 UTC m=+460.950288077" watchObservedRunningTime="2026-07-07 19:16:55.600886835 +0000 UTC m=+460.951535072" Jul 07 19:16:55.629516 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:55.629482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:55.729793 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:55.729760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:56.343400 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:56.343363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:56.523311 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:56.523266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:16:57.590874 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:57.590769 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" event={"ID":"286cf3ec-1384-41a9-905e-bbe14b930996","Type":"ContainerStarted","Data":"945e08709bbc6d7173c9d763866bd9fe94704ece80d038807f8442bc6664505a"} Jul 07 19:16:57.590874 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:57.590849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:16:57.613418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:16:57.613361 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" podStartSLOduration=1.641108485 podStartE2EDuration="4.613339862s" totalImagesPullingTime="2.972231377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:57.610385365 +0000 UTC m=+462.961033616" watchObservedRunningTime="2026-07-07 19:16:57.613339862 +0000 UTC m=+462.963988177" Jul 07 19:17:04.569115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:04.569081 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-57vn5" Jul 07 19:17:04.569562 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:04.569155 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:04.846374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:04.846266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:04.944508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:04.944473 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:06.425044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.425006 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck"] Jul 07 19:17:06.425569 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.425286 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" containerName="manager" containerID="cri-o://48750cdeb784813bd7817c62165e2b9aa696ba4e26c6a431596180c7265d644c" gracePeriod=2 Jul 07 19:17:06.435714 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.435682 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck"] Jul 07 19:17:06.435889 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.435778 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\": pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.446341 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.446305 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb"] Jul 07 19:17:06.446595 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.446568 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" containerName="manager" containerID="cri-o://945e08709bbc6d7173c9d763866bd9fe94704ece80d038807f8442bc6664505a" gracePeriod=2 Jul 07 19:17:06.449139 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.449116 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.449808 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.449783 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:06.450435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.450409 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" containerName="manager" Jul 07 19:17:06.450435 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.450435 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" containerName="manager" Jul 07 19:17:06.450571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.450526 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" containerName="manager" Jul 07 19:17:06.453761 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.453739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.457241 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.457198 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"286cf3ec-1384-41a9-905e-bbe14b930996\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-ss9vb\": pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.458586 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.458554 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb"] Jul 07 19:17:06.459122 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.459082 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.461198 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.461154 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.464538 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.464498 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.464717 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.464668 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:06.471170 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.471142 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp"] Jul 07 19:17:06.471608 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.471586 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" containerName="manager" Jul 07 19:17:06.471608 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.471608 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" containerName="manager" Jul 07 19:17:06.471831 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.471734 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" containerName="manager" Jul 07 19:17:06.475008 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.474984 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:06.484593 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.484544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp"] Jul 07 19:17:06.491986 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.491939 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.493909 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.493864 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.586296 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.586266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-sjr2t" Jul 07 19:17:06.598735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.598704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.598963 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.598801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngvh\" (UniqueName: \"kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.598963 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.598829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2g9t\" (UniqueName: \"kubernetes.io/projected/66984758-bbec-48b1-8422-eb0fcd699464-kube-api-access-g2g9t\") pod \"limitador-operator-controller-manager-859d7b78bd-v48sp\" (UID: \"66984758-bbec-48b1-8422-eb0fcd699464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:06.608482 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.608433 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.610188 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.610154 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.647748 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.647710 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" containerID="48750cdeb784813bd7817c62165e2b9aa696ba4e26c6a431596180c7265d644c" exitCode=0 Jul 07 19:17:06.649603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.649577 2568 generic.go:356] "Generic (PLEG): container finished" podID="286cf3ec-1384-41a9-905e-bbe14b930996" containerID="945e08709bbc6d7173c9d763866bd9fe94704ece80d038807f8442bc6664505a" exitCode=0 Jul 07 19:17:06.699938 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.699684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.700207 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.700152 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.700768 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.699838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngvh\" (UniqueName: \"kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.700768 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.700487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2g9t\" (UniqueName: \"kubernetes.io/projected/66984758-bbec-48b1-8422-eb0fcd699464-kube-api-access-g2g9t\") pod \"limitador-operator-controller-manager-859d7b78bd-v48sp\" (UID: \"66984758-bbec-48b1-8422-eb0fcd699464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:06.716776 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.716723 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2g9t\" (UniqueName: \"kubernetes.io/projected/66984758-bbec-48b1-8422-eb0fcd699464-kube-api-access-g2g9t\") pod \"limitador-operator-controller-manager-859d7b78bd-v48sp\" (UID: \"66984758-bbec-48b1-8422-eb0fcd699464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:06.718172 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.718148 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngvh\" (UniqueName: \"kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7grl2\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.738022 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.737994 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.740460 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.740436 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.740460 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.740445 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.744051 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.744024 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.746331 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.746303 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ss9vb\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:06.751032 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:06.751007 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8z8ck\" is forbidden: User \"system:node:ip-10-0-142-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-116.ec2.internal' and this object" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:06.870605 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.870565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:06.877559 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.877536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:06.901349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.901304 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zjm\" (UniqueName: \"kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm\") pod \"286cf3ec-1384-41a9-905e-bbe14b930996\" (UID: \"286cf3ec-1384-41a9-905e-bbe14b930996\") " Jul 07 19:17:06.901546 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.901417 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume\") pod \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " Jul 07 19:17:06.901620 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.901547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlgv5\" (UniqueName: \"kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5\") pod \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\" (UID: \"aa48c651-7eed-4c94-92f6-8ba7e2a3e03e\") " Jul 07 19:17:06.902092 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.902053 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" (UID: "aa48c651-7eed-4c94-92f6-8ba7e2a3e03e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:06.903967 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.903941 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm" (OuterVolumeSpecName: "kube-api-access-56zjm") pod "286cf3ec-1384-41a9-905e-bbe14b930996" (UID: "286cf3ec-1384-41a9-905e-bbe14b930996"). InnerVolumeSpecName "kube-api-access-56zjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:06.904169 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:06.904144 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5" (OuterVolumeSpecName: "kube-api-access-mlgv5") pod "aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" (UID: "aa48c651-7eed-4c94-92f6-8ba7e2a3e03e"). InnerVolumeSpecName "kube-api-access-mlgv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:07.002701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.002659 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlgv5\" (UniqueName: \"kubernetes.io/projected/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-kube-api-access-mlgv5\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.002701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.002704 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zjm\" (UniqueName: \"kubernetes.io/projected/286cf3ec-1384-41a9-905e-bbe14b930996-kube-api-access-56zjm\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.002935 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.002720 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e-extensions-socket-volume\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:17:07.042495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.042457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:07.042736 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:07.042714 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a62c924_62cc_4a05_a018_8a8836ee30a1.slice/crio-5cd17dbb7218e5c676b5be19e0348491c5c4a96936d29b29294596265c6c6269 WatchSource:0}: Error finding container 5cd17dbb7218e5c676b5be19e0348491c5c4a96936d29b29294596265c6c6269: Status 404 returned error can't find the container with id 5cd17dbb7218e5c676b5be19e0348491c5c4a96936d29b29294596265c6c6269 Jul 07 19:17:07.063113 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.062963 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp"] Jul 07 19:17:07.065881 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:07.065848 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66984758_bbec_48b1_8422_eb0fcd699464.slice/crio-413e282da01f43c7a1e2cf2042ce6ccac8def1178ae20358c4ed7b31b21d36b2 WatchSource:0}: Error finding container 413e282da01f43c7a1e2cf2042ce6ccac8def1178ae20358c4ed7b31b21d36b2: Status 404 returned error can't find the container with id 413e282da01f43c7a1e2cf2042ce6ccac8def1178ae20358c4ed7b31b21d36b2 Jul 07 19:17:07.354328 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.354248 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="286cf3ec-1384-41a9-905e-bbe14b930996" path="/var/lib/kubelet/pods/286cf3ec-1384-41a9-905e-bbe14b930996/volumes" Jul 07 19:17:07.354653 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.354639 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa48c651-7eed-4c94-92f6-8ba7e2a3e03e" path="/var/lib/kubelet/pods/aa48c651-7eed-4c94-92f6-8ba7e2a3e03e/volumes" Jul 07 19:17:07.455713 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.455672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:07.625686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.625652 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:07.657780 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.657742 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" event={"ID":"2a62c924-62cc-4a05-a018-8a8836ee30a1","Type":"ContainerStarted","Data":"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6"} Jul 07 19:17:07.658024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.657793 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" event={"ID":"2a62c924-62cc-4a05-a018-8a8836ee30a1","Type":"ContainerStarted","Data":"5cd17dbb7218e5c676b5be19e0348491c5c4a96936d29b29294596265c6c6269"} Jul 07 19:17:07.658024 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.657837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:07.659317 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.659295 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8z8ck" Jul 07 19:17:07.659418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.659301 2568 scope.go:117] "RemoveContainer" containerID="48750cdeb784813bd7817c62165e2b9aa696ba4e26c6a431596180c7265d644c" Jul 07 19:17:07.660881 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.660732 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ss9vb" Jul 07 19:17:07.662653 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.662625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" event={"ID":"66984758-bbec-48b1-8422-eb0fcd699464","Type":"ContainerStarted","Data":"2f2957bc33f0e0f22f09760651dbe569f7667dabc325f5e0a28b6dedf9e3729c"} Jul 07 19:17:07.662775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.662687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" event={"ID":"66984758-bbec-48b1-8422-eb0fcd699464","Type":"ContainerStarted","Data":"413e282da01f43c7a1e2cf2042ce6ccac8def1178ae20358c4ed7b31b21d36b2"} Jul 07 19:17:07.662837 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.662790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:07.669617 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.669598 2568 scope.go:117] "RemoveContainer" containerID="945e08709bbc6d7173c9d763866bd9fe94704ece80d038807f8442bc6664505a" Jul 07 19:17:07.694252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.694195 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" podStartSLOduration=1.694174135 podStartE2EDuration="1.694174135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:07.692603476 +0000 UTC m=+473.043251733" watchObservedRunningTime="2026-07-07 19:17:07.694174135 +0000 UTC m=+473.044822482" Jul 07 19:17:07.729594 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:07.729531 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" podStartSLOduration=1.729517547 podStartE2EDuration="1.729517547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:07.727286966 +0000 UTC m=+473.077935202" watchObservedRunningTime="2026-07-07 19:17:07.729517547 +0000 UTC m=+473.080165784" Jul 07 19:17:12.045694 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:12.045657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:12.146070 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:12.146032 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:15.841484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:15.841453 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:15.907207 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:15.907168 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:18.673719 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:18.673686 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-v48sp" Jul 07 19:17:18.674286 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:18.673768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:24.308192 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:24.308152 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:24.381553 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:24.381523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:24.620237 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:24.620202 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:24.718701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:24.718668 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:27.072557 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:27.072526 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:27.170600 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:27.170565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:31.894385 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:31.894349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:31.994515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:31.994477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:34.994702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:34.994664 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:35.095311 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:35.095282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:35.162997 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:35.162962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:35.830073 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:35.830032 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:35.897558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:35.897515 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:36.804769 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:36.804728 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:36.805227 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:36.805012 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" podUID="2a62c924-62cc-4a05-a018-8a8836ee30a1" containerName="manager" containerID="cri-o://c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6" gracePeriod=10 Jul 07 19:17:37.056400 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.056325 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:37.177863 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.177826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume\") pod \"2a62c924-62cc-4a05-a018-8a8836ee30a1\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " Jul 07 19:17:37.178068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.177904 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bngvh\" (UniqueName: \"kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh\") pod \"2a62c924-62cc-4a05-a018-8a8836ee30a1\" (UID: \"2a62c924-62cc-4a05-a018-8a8836ee30a1\") " Jul 07 19:17:37.178335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.178301 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a62c924-62cc-4a05-a018-8a8836ee30a1" (UID: "2a62c924-62cc-4a05-a018-8a8836ee30a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:17:37.180010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.179982 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh" (OuterVolumeSpecName: "kube-api-access-bngvh") pod "2a62c924-62cc-4a05-a018-8a8836ee30a1" (UID: "2a62c924-62cc-4a05-a018-8a8836ee30a1"). InnerVolumeSpecName "kube-api-access-bngvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:17:37.278757 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.278715 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a62c924-62cc-4a05-a018-8a8836ee30a1-extensions-socket-volume\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:17:37.278757 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.278748 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngvh\" (UniqueName: \"kubernetes.io/projected/2a62c924-62cc-4a05-a018-8a8836ee30a1-kube-api-access-bngvh\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:17:37.844094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.844010 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a62c924-62cc-4a05-a018-8a8836ee30a1" containerID="c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6" exitCode=0 Jul 07 19:17:37.844094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.844056 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" Jul 07 19:17:37.844527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.844097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" event={"ID":"2a62c924-62cc-4a05-a018-8a8836ee30a1","Type":"ContainerDied","Data":"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6"} Jul 07 19:17:37.844527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.844155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2" event={"ID":"2a62c924-62cc-4a05-a018-8a8836ee30a1","Type":"ContainerDied","Data":"5cd17dbb7218e5c676b5be19e0348491c5c4a96936d29b29294596265c6c6269"} Jul 07 19:17:37.844527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.844183 2568 scope.go:117] "RemoveContainer" containerID="c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6" Jul 07 19:17:37.853572 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.853552 2568 scope.go:117] "RemoveContainer" containerID="c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6" Jul 07 19:17:37.853908 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:17:37.853875 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6\": container with ID starting with c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6 not found: ID does not exist" containerID="c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6" Jul 07 19:17:37.854042 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.853905 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6"} err="failed to get container status \"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6\": rpc error: code = NotFound desc = could not find container \"c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6\": container with ID starting with c0c7b2d8789390840fcd4420c531dadf40e6e6b2f0ca8a8384156ece69792ce6 not found: ID does not exist" Jul 07 19:17:37.865172 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.865130 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:37.873147 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:37.873113 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7grl2"] Jul 07 19:17:39.353076 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:39.353042 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a62c924-62cc-4a05-a018-8a8836ee30a1" path="/var/lib/kubelet/pods/2a62c924-62cc-4a05-a018-8a8836ee30a1/volumes" Jul 07 19:17:46.019254 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:46.019222 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:46.087570 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:46.087530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:46.225334 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:46.225288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:46.306171 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:46.306066 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:47.809060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:47.809019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:47.909446 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:47.909406 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:52.247514 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:52.247477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:52.348255 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:52.348208 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:53.113883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.113839 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g"] Jul 07 19:17:53.114344 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.114325 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a62c924-62cc-4a05-a018-8a8836ee30a1" containerName="manager" Jul 07 19:17:53.114443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.114347 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a62c924-62cc-4a05-a018-8a8836ee30a1" containerName="manager" Jul 07 19:17:53.114443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.114439 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a62c924-62cc-4a05-a018-8a8836ee30a1" containerName="manager" Jul 07 19:17:53.117374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.117350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.119790 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.119764 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv5t5\"" Jul 07 19:17:53.124035 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.124007 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g"] Jul 07 19:17:53.165391 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.165357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:53.226414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226630 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226731 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226731 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc95\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-kube-api-access-mpc95\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.226796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.226781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.327850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.327850 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.327990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc95\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-kube-api-access-mpc95\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328428 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328784 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328458 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328784 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328498 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328784 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.328784 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328614 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.329010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.328861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.330420 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.330398 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.330803 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.330783 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.333640 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.333608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:53.337066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.337038 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc95\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-kube-api-access-mpc95\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.337214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.337196 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00a4dcf-b3fd-4a2b-bf94-bb96feda2147-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qzk2g\" (UID: \"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.429714 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.429627 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:53.583657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.583629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g"] Jul 07 19:17:53.585761 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:53.585732 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00a4dcf_b3fd_4a2b_bf94_bb96feda2147.slice/crio-acd3706202fa7e01afecce2ec918195cf43bb2d924dd1a5f4c8d088a12d85d23 WatchSource:0}: Error finding container acd3706202fa7e01afecce2ec918195cf43bb2d924dd1a5f4c8d088a12d85d23: Status 404 returned error can't find the container with id acd3706202fa7e01afecce2ec918195cf43bb2d924dd1a5f4c8d088a12d85d23 Jul 07 19:17:53.587941 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.587893 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:17:53.588044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.587976 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:17:53.588044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.588024 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:17:53.933703 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.933663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" event={"ID":"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147","Type":"ContainerStarted","Data":"4ae6baa47684ed3e85799e33b04058e0e305567e80d05ea3f33739648d9b1b79"} Jul 07 19:17:53.933897 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.933729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" event={"ID":"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147","Type":"ContainerStarted","Data":"acd3706202fa7e01afecce2ec918195cf43bb2d924dd1a5f4c8d088a12d85d23"} Jul 07 19:17:53.957470 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:53.957412 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" podStartSLOduration=0.957391327 podStartE2EDuration="957.391327ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:53.955766367 +0000 UTC m=+519.306414629" watchObservedRunningTime="2026-07-07 19:17:53.957391327 +0000 UTC m=+519.308039564" Jul 07 19:17:54.430517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:54.430480 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:54.435547 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:54.435519 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:54.939241 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:54.939210 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:54.940294 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:54.940275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:17:55.651582 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.651546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:55.656343 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.656307 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:17:55.659260 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.659236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.661740 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.661718 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lcbmm\"" Jul 07 19:17:55.661872 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.661728 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 19:17:55.668894 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.668867 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:17:55.748729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.748693 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:17:55.750699 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.750671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.750855 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.750807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrrl\" (UniqueName: \"kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.756291 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.756261 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:55.851676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.851637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrrl\" (UniqueName: \"kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.852046 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.852012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.852785 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.852746 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.856796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.856764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:55.860966 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.860937 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrrl\" (UniqueName: \"kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl\") pod \"limitador-limitador-64c94fd64f-2swlr\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:55.971684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:55.971598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:17:56.015230 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.015194 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:17:56.018645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.018618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.030345 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.030287 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:17:56.062991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.062957 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:17:56.124237 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.124210 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:17:56.125253 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:56.125222 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8f3d1b_a6dd_452b_9515_b5e25846c3a6.slice/crio-32c6725f63df61fe8a42843925dfe045b5f635ff90a2e339ff13b51fc83678d0 WatchSource:0}: Error finding container 32c6725f63df61fe8a42843925dfe045b5f635ff90a2e339ff13b51fc83678d0: Status 404 returned error can't find the container with id 32c6725f63df61fe8a42843925dfe045b5f635ff90a2e339ff13b51fc83678d0 Jul 07 19:17:56.153997 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.153945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/631ca361-3284-4640-abcf-47cf8ee18053-config-file\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.154185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.154101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6brh\" (UniqueName: \"kubernetes.io/projected/631ca361-3284-4640-abcf-47cf8ee18053-kube-api-access-s6brh\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.255702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.255605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/631ca361-3284-4640-abcf-47cf8ee18053-config-file\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.255864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.255755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6brh\" (UniqueName: \"kubernetes.io/projected/631ca361-3284-4640-abcf-47cf8ee18053-kube-api-access-s6brh\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.256298 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.256277 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/631ca361-3284-4640-abcf-47cf8ee18053-config-file\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.265173 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.265144 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6brh\" (UniqueName: \"kubernetes.io/projected/631ca361-3284-4640-abcf-47cf8ee18053-kube-api-access-s6brh\") pod \"limitador-limitador-54d6d988d5-j9c2k\" (UID: \"631ca361-3284-4640-abcf-47cf8ee18053\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.335093 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.335061 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:56.486233 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.486201 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:17:56.487857 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:56.487824 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631ca361_3284_4640_abcf_47cf8ee18053.slice/crio-5f007ff712137a3e481eceeae36aa8c0690ebb13fd6f7b377caa0c3c30661366 WatchSource:0}: Error finding container 5f007ff712137a3e481eceeae36aa8c0690ebb13fd6f7b377caa0c3c30661366: Status 404 returned error can't find the container with id 5f007ff712137a3e481eceeae36aa8c0690ebb13fd6f7b377caa0c3c30661366 Jul 07 19:17:56.541537 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.541464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:56.642967 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.642830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:56.744749 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.744707 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:17:56.749055 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.749030 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:17:56.751781 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.751761 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nvfkw\"" Jul 07 19:17:56.756795 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.756434 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:17:56.864611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.864566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4gt\" (UniqueName: \"kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt\") pod \"authorino-75d556bd77-q2x6w\" (UID: \"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4\") " pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:17:56.958534 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.958096 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" event={"ID":"631ca361-3284-4640-abcf-47cf8ee18053","Type":"ContainerStarted","Data":"5f007ff712137a3e481eceeae36aa8c0690ebb13fd6f7b377caa0c3c30661366"} Jul 07 19:17:56.963042 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.962953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" event={"ID":"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6","Type":"ContainerStarted","Data":"32c6725f63df61fe8a42843925dfe045b5f635ff90a2e339ff13b51fc83678d0"} Jul 07 19:17:56.965356 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.965324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4gt\" (UniqueName: \"kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt\") pod \"authorino-75d556bd77-q2x6w\" (UID: \"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4\") " pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:17:56.983983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:56.983836 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4gt\" (UniqueName: \"kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt\") pod \"authorino-75d556bd77-q2x6w\" (UID: \"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4\") " pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:17:57.067763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:57.067730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:17:57.296953 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:57.296873 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:17:57.303193 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:17:57.303112 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd348dbdf_6d20_409c_9e66_a9bbbe8d0ae4.slice/crio-03654752e4fa9577c64691d1d80f5d79c90c52eb46a41387148e01284f56092f WatchSource:0}: Error finding container 03654752e4fa9577c64691d1d80f5d79c90c52eb46a41387148e01284f56092f: Status 404 returned error can't find the container with id 03654752e4fa9577c64691d1d80f5d79c90c52eb46a41387148e01284f56092f Jul 07 19:17:57.970555 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:57.970513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2x6w" event={"ID":"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4","Type":"ContainerStarted","Data":"03654752e4fa9577c64691d1d80f5d79c90c52eb46a41387148e01284f56092f"} Jul 07 19:17:59.873615 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:59.873577 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:17:59.990501 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:59.990469 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" event={"ID":"631ca361-3284-4640-abcf-47cf8ee18053","Type":"ContainerStarted","Data":"1b3f66eae5521b70928b3e28d84f82bd812936df7beeffbece8007ca280e5889"} Jul 07 19:17:59.990688 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:59.990584 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:17:59.992405 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:59.992375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" event={"ID":"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6","Type":"ContainerStarted","Data":"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9"} Jul 07 19:17:59.992552 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:17:59.992458 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:18:00.024137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:00.024067 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" podStartSLOduration=2.325191292 podStartE2EDuration="5.024045863s" totalImagesPullingTime="2.698854571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:00.021131094 +0000 UTC m=+525.371779369" watchObservedRunningTime="2026-07-07 19:18:00.024045863 +0000 UTC m=+525.374694101" Jul 07 19:18:00.044093 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:00.044038 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" podStartSLOduration=1.9770721070000001 podStartE2EDuration="5.044021152s" totalImagesPullingTime="3.066949045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:00.041481743 +0000 UTC m=+525.392130001" watchObservedRunningTime="2026-07-07 19:18:00.044021152 +0000 UTC m=+525.394669393" Jul 07 19:18:02.008408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:02.008368 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2x6w" event={"ID":"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4","Type":"ContainerStarted","Data":"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555"} Jul 07 19:18:02.027462 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:02.027402 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q2x6w" podStartSLOduration=2.258348195 podStartE2EDuration="6.027384509s" totalImagesPullingTime="3.769036314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:02.024893442 +0000 UTC m=+527.375541680" watchObservedRunningTime="2026-07-07 19:18:02.027384509 +0000 UTC m=+527.378032775" Jul 07 19:18:10.968365 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:10.968282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:11.000976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.000906 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-j9c2k" Jul 07 19:18:11.001120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.001029 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:18:11.087813 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.087778 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:18:11.088013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.087993 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" podUID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" containerName="limitador" containerID="cri-o://0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9" gracePeriod=30 Jul 07 19:18:11.642870 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.642840 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:18:11.707419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.707324 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lrrl\" (UniqueName: \"kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl\") pod \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " Jul 07 19:18:11.707419 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.707398 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file\") pod \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\" (UID: \"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6\") " Jul 07 19:18:11.707775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.707747 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file" (OuterVolumeSpecName: "config-file") pod "9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" (UID: "9c8f3d1b-a6dd-452b-9515-b5e25846c3a6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:18:11.709492 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.709464 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl" (OuterVolumeSpecName: "kube-api-access-9lrrl") pod "9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" (UID: "9c8f3d1b-a6dd-452b-9515-b5e25846c3a6"). InnerVolumeSpecName "kube-api-access-9lrrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:18:11.808614 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.808577 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrrl\" (UniqueName: \"kubernetes.io/projected/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-kube-api-access-9lrrl\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:18:11.808614 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:11.808608 2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6-config-file\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:18:12.067325 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.067232 2568 generic.go:356] "Generic (PLEG): container finished" podID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" containerID="0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9" exitCode=0 Jul 07 19:18:12.067325 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.067280 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" Jul 07 19:18:12.067325 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.067317 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" event={"ID":"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6","Type":"ContainerDied","Data":"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9"} Jul 07 19:18:12.067815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.067374 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2swlr" event={"ID":"9c8f3d1b-a6dd-452b-9515-b5e25846c3a6","Type":"ContainerDied","Data":"32c6725f63df61fe8a42843925dfe045b5f635ff90a2e339ff13b51fc83678d0"} Jul 07 19:18:12.067815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.067396 2568 scope.go:117] "RemoveContainer" containerID="0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9" Jul 07 19:18:12.077134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.077113 2568 scope.go:117] "RemoveContainer" containerID="0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9" Jul 07 19:18:12.077483 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:18:12.077462 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9\": container with ID starting with 0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9 not found: ID does not exist" containerID="0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9" Jul 07 19:18:12.077558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.077493 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9"} err="failed to get container status \"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9\": rpc error: code = NotFound desc = could not find container \"0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9\": container with ID starting with 0458b2e91dec46ca1888eb69f0ad256951f54fca2d1afbfb194a301872c25cd9 not found: ID does not exist" Jul 07 19:18:12.095031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.094986 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:18:12.105047 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:12.105014 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2swlr"] Jul 07 19:18:13.353326 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:13.353280 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" path="/var/lib/kubelet/pods/9c8f3d1b-a6dd-452b-9515-b5e25846c3a6/volumes" Jul 07 19:18:16.518502 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:16.518458 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:28.572994 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.572957 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:28.788531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.788491 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:28.788899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.788886 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" containerName="limitador" Jul 07 19:18:28.788899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.788900 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" containerName="limitador" Jul 07 19:18:28.789039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.789027 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8f3d1b-a6dd-452b-9515-b5e25846c3a6" containerName="limitador" Jul 07 19:18:28.797974 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.797942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:28.798793 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.798761 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:28.865704 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.865671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hk7d\" (UniqueName: \"kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d\") pod \"authorino-5d4b67c446-sphz6\" (UID: \"1a61969f-ad95-4f90-bf51-d7e6d545cb0f\") " pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:28.966957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.966902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hk7d\" (UniqueName: \"kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d\") pod \"authorino-5d4b67c446-sphz6\" (UID: \"1a61969f-ad95-4f90-bf51-d7e6d545cb0f\") " pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:28.973153 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.973120 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:28.976041 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:28.976011 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hk7d\" (UniqueName: \"kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d\") pod \"authorino-5d4b67c446-sphz6\" (UID: \"1a61969f-ad95-4f90-bf51-d7e6d545cb0f\") " pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:29.019493 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.019451 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:29.019794 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.019776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:29.169783 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.169759 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:29.171495 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:18:29.171465 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a61969f_ad95_4f90_bf51_d7e6d545cb0f.slice/crio-778cb717fb3610cbfd0908bc468c00d5333d22bde30d53098d045d7cff30fce6 WatchSource:0}: Error finding container 778cb717fb3610cbfd0908bc468c00d5333d22bde30d53098d045d7cff30fce6: Status 404 returned error can't find the container with id 778cb717fb3610cbfd0908bc468c00d5333d22bde30d53098d045d7cff30fce6 Jul 07 19:18:29.458013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.457901 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:18:29.464110 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.464080 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.466656 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.466628 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 19:18:29.476247 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.476219 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:18:29.571789 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.571750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.571994 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.571855 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mr8d\" (UniqueName: \"kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.672934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.672883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.673388 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.673016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mr8d\" (UniqueName: \"kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.675670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.675641 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.683405 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.683369 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mr8d\" (UniqueName: \"kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d\") pod \"authorino-9478b4cfb-4x927\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.775934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.775796 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:18:29.926888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:29.926856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:18:29.928897 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:18:29.928871 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c4b3c49_ddab_428b_9ea2_433f0839c7c3.slice/crio-ff6784a26e59d6eda85269de9ad68735715d2a9a6ef1c4af0f7c6a9cc1929be8 WatchSource:0}: Error finding container ff6784a26e59d6eda85269de9ad68735715d2a9a6ef1c4af0f7c6a9cc1929be8: Status 404 returned error can't find the container with id ff6784a26e59d6eda85269de9ad68735715d2a9a6ef1c4af0f7c6a9cc1929be8 Jul 07 19:18:30.177304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.177261 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9478b4cfb-4x927" event={"ID":"3c4b3c49-ddab-428b-9ea2-433f0839c7c3","Type":"ContainerStarted","Data":"ff6784a26e59d6eda85269de9ad68735715d2a9a6ef1c4af0f7c6a9cc1929be8"} Jul 07 19:18:30.178802 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.178771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sphz6" event={"ID":"1a61969f-ad95-4f90-bf51-d7e6d545cb0f","Type":"ContainerStarted","Data":"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724"} Jul 07 19:18:30.178962 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.178815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sphz6" event={"ID":"1a61969f-ad95-4f90-bf51-d7e6d545cb0f","Type":"ContainerStarted","Data":"778cb717fb3610cbfd0908bc468c00d5333d22bde30d53098d045d7cff30fce6"} Jul 07 19:18:30.178962 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.178869 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sphz6" podUID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" containerName="authorino" containerID="cri-o://bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724" gracePeriod=30 Jul 07 19:18:30.436239 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.436162 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:30.586084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.586039 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hk7d\" (UniqueName: \"kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d\") pod \"1a61969f-ad95-4f90-bf51-d7e6d545cb0f\" (UID: \"1a61969f-ad95-4f90-bf51-d7e6d545cb0f\") " Jul 07 19:18:30.588196 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.588170 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d" (OuterVolumeSpecName: "kube-api-access-9hk7d") pod "1a61969f-ad95-4f90-bf51-d7e6d545cb0f" (UID: "1a61969f-ad95-4f90-bf51-d7e6d545cb0f"). InnerVolumeSpecName "kube-api-access-9hk7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:18:30.686975 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:30.686839 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hk7d\" (UniqueName: \"kubernetes.io/projected/1a61969f-ad95-4f90-bf51-d7e6d545cb0f-kube-api-access-9hk7d\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:18:31.186454 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.186418 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9478b4cfb-4x927" event={"ID":"3c4b3c49-ddab-428b-9ea2-433f0839c7c3","Type":"ContainerStarted","Data":"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562"} Jul 07 19:18:31.187787 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.187762 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" containerID="bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724" exitCode=0 Jul 07 19:18:31.187948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.187799 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sphz6" Jul 07 19:18:31.187948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.187809 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sphz6" event={"ID":"1a61969f-ad95-4f90-bf51-d7e6d545cb0f","Type":"ContainerDied","Data":"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724"} Jul 07 19:18:31.187948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.187837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sphz6" event={"ID":"1a61969f-ad95-4f90-bf51-d7e6d545cb0f","Type":"ContainerDied","Data":"778cb717fb3610cbfd0908bc468c00d5333d22bde30d53098d045d7cff30fce6"} Jul 07 19:18:31.187948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.187856 2568 scope.go:117] "RemoveContainer" containerID="bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724" Jul 07 19:18:31.197042 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.197016 2568 scope.go:117] "RemoveContainer" containerID="bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724" Jul 07 19:18:31.197382 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:18:31.197363 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724\": container with ID starting with bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724 not found: ID does not exist" containerID="bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724" Jul 07 19:18:31.197449 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.197397 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724"} err="failed to get container status \"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724\": rpc error: code = NotFound desc = could not find container \"bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724\": container with ID starting with bfe097276aeac113758fa94f8f84f32dc26c39d2dc0db09deacc410c94de7724 not found: ID does not exist" Jul 07 19:18:31.212488 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.212159 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9478b4cfb-4x927" podStartSLOduration=1.773032999 podStartE2EDuration="2.212135037s" totalImagesPullingTime="439.102038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:31.208842657 +0000 UTC m=+556.559490894" watchObservedRunningTime="2026-07-07 19:18:31.212135037 +0000 UTC m=+556.562783276" Jul 07 19:18:31.230470 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.230436 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:31.234263 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.234176 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sphz6"] Jul 07 19:18:31.257662 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.257626 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:18:31.257849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.257820 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q2x6w" podUID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" containerName="authorino" containerID="cri-o://5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555" gracePeriod=30 Jul 07 19:18:31.353820 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.353784 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" path="/var/lib/kubelet/pods/1a61969f-ad95-4f90-bf51-d7e6d545cb0f/volumes" Jul 07 19:18:31.517111 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.517082 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:18:31.594362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.594327 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw4gt\" (UniqueName: \"kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt\") pod \"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4\" (UID: \"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4\") " Jul 07 19:18:31.596397 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.596362 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt" (OuterVolumeSpecName: "kube-api-access-fw4gt") pod "d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" (UID: "d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4"). InnerVolumeSpecName "kube-api-access-fw4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:18:31.695719 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:31.695685 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4gt\" (UniqueName: \"kubernetes.io/projected/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4-kube-api-access-fw4gt\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:18:32.195339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.195305 2568 generic.go:356] "Generic (PLEG): container finished" podID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" containerID="5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555" exitCode=0 Jul 07 19:18:32.195339 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.195341 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2x6w" Jul 07 19:18:32.195622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.195394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2x6w" event={"ID":"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4","Type":"ContainerDied","Data":"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555"} Jul 07 19:18:32.195622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.195435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2x6w" event={"ID":"d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4","Type":"ContainerDied","Data":"03654752e4fa9577c64691d1d80f5d79c90c52eb46a41387148e01284f56092f"} Jul 07 19:18:32.195622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.195459 2568 scope.go:117] "RemoveContainer" containerID="5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555" Jul 07 19:18:32.205413 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.205391 2568 scope.go:117] "RemoveContainer" containerID="5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555" Jul 07 19:18:32.205744 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:18:32.205722 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555\": container with ID starting with 5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555 not found: ID does not exist" containerID="5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555" Jul 07 19:18:32.205814 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.205757 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555"} err="failed to get container status \"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555\": rpc error: code = NotFound desc = could not find container \"5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555\": container with ID starting with 5d89e7d07b026698c9f1576fbe97e8a852a516eb682dea276001606769691555 not found: ID does not exist" Jul 07 19:18:32.236505 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.236459 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:18:32.247544 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:32.247514 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2x6w"] Jul 07 19:18:33.352590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:33.352553 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" path="/var/lib/kubelet/pods/d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4/volumes" Jul 07 19:18:35.653375 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:35.653325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:35.742955 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:35.742898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:35.817525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:35.817496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:35.959454 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:35.959365 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:38.794741 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:38.794702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:38.895549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:38.895506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:42.841808 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:42.841761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:49.419619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:49.419588 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:18:49.503026 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:18:49.502988 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:10.512308 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:10.512272 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:10.980111 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:10.980067 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:11.009873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.009823 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-69775b867f-jptc7"] Jul 07 19:19:11.011317 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.011289 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" containerName="authorino" Jul 07 19:19:11.011592 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.011578 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" containerName="authorino" Jul 07 19:19:11.011756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.011746 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" containerName="authorino" Jul 07 19:19:11.011830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.011821 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" containerName="authorino" Jul 07 19:19:11.012143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.012128 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a61969f-ad95-4f90-bf51-d7e6d545cb0f" containerName="authorino" Jul 07 19:19:11.012391 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.012377 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d348dbdf-6d20-409c-9e66-a9bbbe8d0ae4" containerName="authorino" Jul 07 19:19:11.017072 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.017041 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.019516 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.019487 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 19:19:11.020944 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.020899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69775b867f-jptc7"] Jul 07 19:19:11.172360 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.172322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkf4m\" (UniqueName: \"kubernetes.io/projected/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-kube-api-access-wkf4m\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.172552 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.172445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-openshift-service-ca\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.172552 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.172509 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.273516 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.273412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-openshift-service-ca\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.273516 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.273480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.273717 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.273548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkf4m\" (UniqueName: \"kubernetes.io/projected/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-kube-api-access-wkf4m\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.273717 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:19:11.273629 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 07 19:19:11.273799 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:19:11.273736 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls podName:ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e nodeName:}" failed. No retries permitted until 2026-07-07 19:19:11.773711434 +0000 UTC m=+597.124359654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls") pod "maas-api-69775b867f-jptc7" (UID: "ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e") : secret "maas-api-serving-cert" not found Jul 07 19:19:11.274226 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.274207 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-openshift-service-ca\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.284711 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.284673 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkf4m\" (UniqueName: \"kubernetes.io/projected/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-kube-api-access-wkf4m\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.593396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.593282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:11.690862 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.690826 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:11.778847 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.778807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.781324 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.781293 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e-maas-api-tls\") pod \"maas-api-69775b867f-jptc7\" (UID: \"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e\") " pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:11.785558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.785524 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:11.864001 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.863962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:19:11.929511 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:11.929475 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:12.094236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:12.094205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69775b867f-jptc7"] Jul 07 19:19:12.095682 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:19:12.095651 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfe8aa0_6656_493b_82f0_a64d3fe0ef5e.slice/crio-2b61492a57ef6d0c79a7c99dc411e03e0cc9f90a8308ccb3f377d0f0f66b813f WatchSource:0}: Error finding container 2b61492a57ef6d0c79a7c99dc411e03e0cc9f90a8308ccb3f377d0f0f66b813f: Status 404 returned error can't find the container with id 2b61492a57ef6d0c79a7c99dc411e03e0cc9f90a8308ccb3f377d0f0f66b813f Jul 07 19:19:12.442456 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:12.442419 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69775b867f-jptc7" event={"ID":"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e","Type":"ContainerStarted","Data":"2b61492a57ef6d0c79a7c99dc411e03e0cc9f90a8308ccb3f377d0f0f66b813f"} Jul 07 19:19:14.471532 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:14.471492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69775b867f-jptc7" event={"ID":"ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e","Type":"ContainerStarted","Data":"cdd1f60142d98a65550847b3ede9c51e7a3dc309e643759c6f6e08f1c89555c6"} Jul 07 19:19:14.471950 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:14.471555 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:14.491631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:14.491574 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-69775b867f-jptc7" podStartSLOduration=2.566182513 podStartE2EDuration="4.491557332s" totalImagesPullingTime="1.925374819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:19:14.489645382 +0000 UTC m=+599.840293674" watchObservedRunningTime="2026-07-07 19:19:14.491557332 +0000 UTC m=+599.842205568" Jul 07 19:19:20.483632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.483600 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-69775b867f-jptc7" Jul 07 19:19:20.510270 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.510233 2568 generic.go:356] "Generic (PLEG): container finished" podID="b00a4dcf-b3fd-4a2b-bf94-bb96feda2147" containerID="4ae6baa47684ed3e85799e33b04058e0e305567e80d05ea3f33739648d9b1b79" exitCode=0 Jul 07 19:19:20.510449 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.510309 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" event={"ID":"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147","Type":"ContainerDied","Data":"4ae6baa47684ed3e85799e33b04058e0e305567e80d05ea3f33739648d9b1b79"} Jul 07 19:19:20.510737 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.510723 2568 scope.go:117] "RemoveContainer" containerID="4ae6baa47684ed3e85799e33b04058e0e305567e80d05ea3f33739648d9b1b79" Jul 07 19:19:20.511819 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.511792 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:19:20.511878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.511856 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:19:20.511932 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:20.511889 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:19:21.517498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:21.517459 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" event={"ID":"b00a4dcf-b3fd-4a2b-bf94-bb96feda2147","Type":"ContainerStarted","Data":"3ae22d11b9f05d44b2da03c33aec388bec763d89c27b68e5305766427536c186"} Jul 07 19:19:21.759631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:21.759589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:21.860947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:21.860893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:22.430277 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:22.430230 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:19:22.431569 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:22.431531 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qzk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:19:22.431714 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:22.431586 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" podUID="b00a4dcf-b3fd-4a2b-bf94-bb96feda2147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:19:23.430927 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:23.430871 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qzk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:19:23.431330 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:23.430980 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" podUID="b00a4dcf-b3fd-4a2b-bf94-bb96feda2147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:19:24.430447 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:24.430408 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qzk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:19:24.430634 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:24.430488 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" podUID="b00a4dcf-b3fd-4a2b-bf94-bb96feda2147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:19:24.939710 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:24.939665 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:19:25.430832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:25.430789 2568 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qzk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:19:25.431120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:25.430847 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" podUID="b00a4dcf-b3fd-4a2b-bf94-bb96feda2147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:19:26.434663 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:26.434632 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:19:26.435625 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:26.435606 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qzk2g" Jul 07 19:19:31.430222 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.430180 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw"] Jul 07 19:19:31.433469 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.433441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.435905 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.435881 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l9d5q\"" Jul 07 19:19:31.436706 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.436681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 19:19:31.436851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.436710 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 19:19:31.436851 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.436716 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 19:19:31.448494 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.448461 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw"] Jul 07 19:19:31.464868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.464835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78193ed-8be6-4792-b06e-6895dda565e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.465066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.464878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.465066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.464973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.465066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.464990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.465066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.465009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsbk\" (UniqueName: \"kubernetes.io/projected/f78193ed-8be6-4792-b06e-6895dda565e7-kube-api-access-jtsbk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.465231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.465180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.496857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.496819 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:31.566357 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566357 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78193ed-8be6-4792-b06e-6895dda565e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566622 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsbk\" (UniqueName: \"kubernetes.io/projected/f78193ed-8be6-4792-b06e-6895dda565e7-kube-api-access-jtsbk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.566891 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566864 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.567039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566895 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.567039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.566971 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.568681 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.568652 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78193ed-8be6-4792-b06e-6895dda565e7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.568948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.568907 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78193ed-8be6-4792-b06e-6895dda565e7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.577858 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.577820 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsbk\" (UniqueName: \"kubernetes.io/projected/f78193ed-8be6-4792-b06e-6895dda565e7-kube-api-access-jtsbk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw\" (UID: \"f78193ed-8be6-4792-b06e-6895dda565e7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.745653 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.745551 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:31.909327 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.909275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw"] Jul 07 19:19:31.910882 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:19:31.910852 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78193ed_8be6_4792_b06e_6895dda565e7.slice/crio-1d67206be05e2be876d4777952ee05fac372fb87606d0db06a7e7995bc058f97 WatchSource:0}: Error finding container 1d67206be05e2be876d4777952ee05fac372fb87606d0db06a7e7995bc058f97: Status 404 returned error can't find the container with id 1d67206be05e2be876d4777952ee05fac372fb87606d0db06a7e7995bc058f97 Jul 07 19:19:31.912375 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.912350 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:19:31.912500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:31.912430 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:19:32.599473 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:32.599421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" event={"ID":"f78193ed-8be6-4792-b06e-6895dda565e7","Type":"ContainerStarted","Data":"1d67206be05e2be876d4777952ee05fac372fb87606d0db06a7e7995bc058f97"} Jul 07 19:19:33.606572 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:33.606516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" event={"ID":"f78193ed-8be6-4792-b06e-6895dda565e7","Type":"ContainerStarted","Data":"c47afe5a0ca59650c31d41da0f560d7b2d5bf19febad0fb9a14fe1dddc238206"} Jul 07 19:19:33.629286 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:33.629229 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" podStartSLOduration=1.563442653 podStartE2EDuration="2.629210996s" totalImagesPullingTime="1.065768343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:19:33.626574859 +0000 UTC m=+618.977223116" watchObservedRunningTime="2026-07-07 19:19:33.629210996 +0000 UTC m=+618.979859232" Jul 07 19:19:41.746475 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:41.746381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:41.746976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:41.746709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:41.759353 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:41.759322 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:42.670729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:42.670700 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw" Jul 07 19:19:44.058668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:44.058626 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:44.802660 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:44.802622 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:19:50.919568 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:50.919529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:52.203049 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:52.203006 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:19:58.367605 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:58.367568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:19:59.708445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:19:59.708389 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:20:14.660548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:20:14.660510 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:16.807209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:20:16.807171 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:20:26.111134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:20:26.111099 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:20:28.822366 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:20:28.822327 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:21:16.900774 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:21:16.900733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:21:33.426787 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:21:33.426745 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:06.307517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.307471 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-658c6b66d5-2bvg9"] Jul 07 19:22:06.311235 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.311209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.321032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.320999 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658c6b66d5-2bvg9"] Jul 07 19:22:06.451985 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.451948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9440065-e959-4c0b-a0cd-8015de89e0e8-tls-cert\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.452175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.452033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwvj\" (UniqueName: \"kubernetes.io/projected/e9440065-e959-4c0b-a0cd-8015de89e0e8-kube-api-access-rxwvj\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.530307 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.530259 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:06.553566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.553521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwvj\" (UniqueName: \"kubernetes.io/projected/e9440065-e959-4c0b-a0cd-8015de89e0e8-kube-api-access-rxwvj\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.553717 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.553623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9440065-e959-4c0b-a0cd-8015de89e0e8-tls-cert\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.556320 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.556281 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9440065-e959-4c0b-a0cd-8015de89e0e8-tls-cert\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.565215 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.565135 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwvj\" (UniqueName: \"kubernetes.io/projected/e9440065-e959-4c0b-a0cd-8015de89e0e8-kube-api-access-rxwvj\") pod \"authorino-658c6b66d5-2bvg9\" (UID: \"e9440065-e959-4c0b-a0cd-8015de89e0e8\") " pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.621849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.621811 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658c6b66d5-2bvg9" Jul 07 19:22:06.624047 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.624021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:06.768645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:06.768606 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658c6b66d5-2bvg9"] Jul 07 19:22:06.770349 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:22:06.770313 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9440065_e959_4c0b_a0cd_8015de89e0e8.slice/crio-490eff9fe9f6f42f1e30c074edc44fbb8cef5455cf3bd6e553c17b348c7dff19 WatchSource:0}: Error finding container 490eff9fe9f6f42f1e30c074edc44fbb8cef5455cf3bd6e553c17b348c7dff19: Status 404 returned error can't find the container with id 490eff9fe9f6f42f1e30c074edc44fbb8cef5455cf3bd6e553c17b348c7dff19 Jul 07 19:22:07.523633 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.523534 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658c6b66d5-2bvg9" event={"ID":"e9440065-e959-4c0b-a0cd-8015de89e0e8","Type":"ContainerStarted","Data":"240af198680ea405d9053891b17121a7c2f132de1412d7c1789c9b66ffa3cac9"} Jul 07 19:22:07.523633 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.523584 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658c6b66d5-2bvg9" event={"ID":"e9440065-e959-4c0b-a0cd-8015de89e0e8","Type":"ContainerStarted","Data":"490eff9fe9f6f42f1e30c074edc44fbb8cef5455cf3bd6e553c17b348c7dff19"} Jul 07 19:22:07.550566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.550491 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-658c6b66d5-2bvg9" podStartSLOduration=1.085976302 podStartE2EDuration="1.550439238s" totalImagesPullingTime="464.462936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:22:07.549409553 +0000 UTC m=+772.900057790" watchObservedRunningTime="2026-07-07 19:22:07.550439238 +0000 UTC m=+772.901087473" Jul 07 19:22:07.603512 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.603473 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:22:07.603985 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.603951 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-9478b4cfb-4x927" podUID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" containerName="authorino" containerID="cri-o://c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562" gracePeriod=30 Jul 07 19:22:07.641981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.641932 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:07.774486 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.774323 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:22:07.894931 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.894888 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:22:07.966599 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.966561 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mr8d\" (UniqueName: \"kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d\") pod \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " Jul 07 19:22:07.966756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.966743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert\") pod \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\" (UID: \"3c4b3c49-ddab-428b-9ea2-433f0839c7c3\") " Jul 07 19:22:07.968683 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.968648 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d" (OuterVolumeSpecName: "kube-api-access-2mr8d") pod "3c4b3c49-ddab-428b-9ea2-433f0839c7c3" (UID: "3c4b3c49-ddab-428b-9ea2-433f0839c7c3"). InnerVolumeSpecName "kube-api-access-2mr8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:22:07.979251 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:07.979219 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3c4b3c49-ddab-428b-9ea2-433f0839c7c3" (UID: "3c4b3c49-ddab-428b-9ea2-433f0839c7c3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:22:08.068068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.067966 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mr8d\" (UniqueName: \"kubernetes.io/projected/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-kube-api-access-2mr8d\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:22:08.068068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.067997 2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c4b3c49-ddab-428b-9ea2-433f0839c7c3-tls-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:22:08.532303 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.532260 2568 generic.go:356] "Generic (PLEG): container finished" podID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" containerID="c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562" exitCode=0 Jul 07 19:22:08.532303 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.532300 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9478b4cfb-4x927" Jul 07 19:22:08.532806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.532344 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9478b4cfb-4x927" event={"ID":"3c4b3c49-ddab-428b-9ea2-433f0839c7c3","Type":"ContainerDied","Data":"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562"} Jul 07 19:22:08.532806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.532392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9478b4cfb-4x927" event={"ID":"3c4b3c49-ddab-428b-9ea2-433f0839c7c3","Type":"ContainerDied","Data":"ff6784a26e59d6eda85269de9ad68735715d2a9a6ef1c4af0f7c6a9cc1929be8"} Jul 07 19:22:08.532806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.532414 2568 scope.go:117] "RemoveContainer" containerID="c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562" Jul 07 19:22:08.542942 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.542901 2568 scope.go:117] "RemoveContainer" containerID="c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562" Jul 07 19:22:08.543240 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:22:08.543218 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562\": container with ID starting with c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562 not found: ID does not exist" containerID="c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562" Jul 07 19:22:08.543326 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.543250 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562"} err="failed to get container status \"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562\": rpc error: code = NotFound desc = could not find container \"c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562\": container with ID starting with c876d3fc23de5543ab41f8a480c1044b3928055702356845b6da78591e44e562 not found: ID does not exist" Jul 07 19:22:08.556374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.556320 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:22:08.560680 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:08.560643 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9478b4cfb-4x927"] Jul 07 19:22:09.352842 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:09.352808 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" path="/var/lib/kubelet/pods/3c4b3c49-ddab-428b-9ea2-433f0839c7c3/volumes" Jul 07 19:22:33.921822 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:33.921779 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:22:46.413781 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:46.413698 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:22:55.014012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:22:55.013970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:23:04.809527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:04.809489 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:23:17.706038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:17.705997 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:23:18.032546 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:18.032456 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:20.115149 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:20.115107 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:22.297796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:22.297761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:22.374167 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:22.374128 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:23:43.916230 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:43.916191 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:23:54.709383 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:23:54.709343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:24:02.971243 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:02.971204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:03.108336 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:03.108298 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:03.238705 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:03.238606 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:10.156676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:10.156580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:10.284368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:10.284330 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:10.406119 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:10.406084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:10.548145 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:10.548049 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:17.597817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:17.597770 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:17.718333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:17.718294 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:24:33.411471 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:33.411424 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:24:42.311231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:42.311187 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:24:51.509464 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:51.509416 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:24:57.320593 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:24:57.320543 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:25:08.319869 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:08.319827 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:25:17.242366 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:17.242317 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:25:22.205674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:22.205632 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:29.007121 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:29.007074 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:25:32.792549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:32.792505 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:32.893144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:32.893103 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:25:38.513069 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:25:38.512977 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:26:00.816754 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:26:00.816714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:26:07.424146 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:26:07.424109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:26:10.206105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:26:10.206063 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:26:35.516732 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:26:35.516694 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:26:47.324896 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:26:47.324856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:27:02.313409 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:27:02.313334 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:27:11.616690 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:27:11.616652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:27:22.206447 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:27:22.206405 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:27:31.513321 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:27:31.513276 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:27:40.115207 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:27:40.115161 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:28:53.370318 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:28:53.370233 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:20.514467 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:29:20.514430 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:20.613276 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:29:20.613243 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:29:28.519247 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:29:28.519205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:29:37.563991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:29:37.563952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:29:51.476589 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:29:51.476542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:00.158812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.158768 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-xwjv2"] Jul 07 19:30:00.159362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.159238 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" containerName="authorino" Jul 07 19:30:00.159362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.159252 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" containerName="authorino" Jul 07 19:30:00.159362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.159340 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c4b3c49-ddab-428b-9ea2-433f0839c7c3" containerName="authorino" Jul 07 19:30:00.162553 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.162532 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:00.165172 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.165146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8pz6d\"" Jul 07 19:30:00.173899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.173868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-xwjv2"] Jul 07 19:30:00.227827 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.227785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgf6\" (UniqueName: \"kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6\") pod \"maas-api-key-cleanup-29724210-xwjv2\" (UID: \"7aab2161-05dc-4edd-bc8c-bd89b6941cf4\") " pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:00.329150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.329095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgf6\" (UniqueName: \"kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6\") pod \"maas-api-key-cleanup-29724210-xwjv2\" (UID: \"7aab2161-05dc-4edd-bc8c-bd89b6941cf4\") " pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:00.339632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.339591 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgf6\" (UniqueName: \"kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6\") pod \"maas-api-key-cleanup-29724210-xwjv2\" (UID: \"7aab2161-05dc-4edd-bc8c-bd89b6941cf4\") " pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:00.474717 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.474628 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:00.624673 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.624643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724210-xwjv2"] Jul 07 19:30:00.625815 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:30:00.625774 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aab2161_05dc_4edd_bc8c_bd89b6941cf4.slice/crio-8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7 WatchSource:0}: Error finding container 8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7: Status 404 returned error can't find the container with id 8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7 Jul 07 19:30:00.627698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:00.627682 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:30:01.310433 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:01.310389 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" event={"ID":"7aab2161-05dc-4edd-bc8c-bd89b6941cf4","Type":"ContainerStarted","Data":"8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7"} Jul 07 19:30:02.021532 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:02.021493 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:04.336421 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:04.336334 2568 generic.go:356] "Generic (PLEG): container finished" podID="7aab2161-05dc-4edd-bc8c-bd89b6941cf4" containerID="f096469b0aa314001bec9b2fca47b80add9f7642fe6cabd58a07b2c5607b694f" exitCode=0 Jul 07 19:30:04.336421 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:04.336400 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" event={"ID":"7aab2161-05dc-4edd-bc8c-bd89b6941cf4","Type":"ContainerDied","Data":"f096469b0aa314001bec9b2fca47b80add9f7642fe6cabd58a07b2c5607b694f"} Jul 07 19:30:05.485619 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:05.485590 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:05.583437 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:05.583391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxgf6\" (UniqueName: \"kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6\") pod \"7aab2161-05dc-4edd-bc8c-bd89b6941cf4\" (UID: \"7aab2161-05dc-4edd-bc8c-bd89b6941cf4\") " Jul 07 19:30:05.585686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:05.585659 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6" (OuterVolumeSpecName: "kube-api-access-wxgf6") pod "7aab2161-05dc-4edd-bc8c-bd89b6941cf4" (UID: "7aab2161-05dc-4edd-bc8c-bd89b6941cf4"). InnerVolumeSpecName "kube-api-access-wxgf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:30:05.684250 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:05.684221 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxgf6\" (UniqueName: \"kubernetes.io/projected/7aab2161-05dc-4edd-bc8c-bd89b6941cf4-kube-api-access-wxgf6\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:30:06.355627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:06.355541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" event={"ID":"7aab2161-05dc-4edd-bc8c-bd89b6941cf4","Type":"ContainerDied","Data":"8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7"} Jul 07 19:30:06.355627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:06.355573 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724210-xwjv2" Jul 07 19:30:06.355627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:06.355617 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0f2ed91a5b2bd9ba532248bedfb0ff3edb9e73995bf6ad1118e9c63a31dcf7" Jul 07 19:30:12.415347 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:12.415299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:24.017772 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:24.017728 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:32.712607 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:32.712563 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:43.420929 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:43.420861 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:30:54.707997 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:30:54.707958 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:31:04.105443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:31:04.105400 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:31:13.726225 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:31:13.726186 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:31:38.508531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:31:38.508439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:31:45.725553 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:31:45.725510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:31:55.405878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:31:55.405832 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:05.919053 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:05.919012 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:13.234284 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:13.234242 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:24.407629 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:24.407588 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:33.412519 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:33.412469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:42.415487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:42.415443 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:32:46.507032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:46.507002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:32:52.207110 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:32:52.207069 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:33:00.707051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:33:00.707011 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:33:09.323236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:33:09.323148 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:33:41.701229 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:33:41.701193 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:33:48.714798 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:33:48.714745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:33:57.509678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:33:57.509635 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:34:06.908175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:34:06.908130 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:34:20.924369 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:34:20.924323 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:34:32.113283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:34:32.113241 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:34:35.384518 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:34:35.384485 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:34:48.403899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:34:48.403853 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:00.104046 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:00.104002 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:12.105957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:12.105901 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:13.832617 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:13.832582 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:13.930217 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:13.930182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:21.264198 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:21.264163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:24.015594 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:24.015544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:29.514711 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:29.514680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:32.811087 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:32.811038 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:34.648099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:34.648068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:37.411829 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:37.411792 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:37.509843 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:37.509808 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:37.854857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:37.854768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:43.802404 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:43.802331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:44.437221 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.437182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:44.539064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.539025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:44.622262 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.622229 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:44.622729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.622711 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aab2161-05dc-4edd-bc8c-bd89b6941cf4" containerName="cleanup" Jul 07 19:35:44.622824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.622732 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aab2161-05dc-4edd-bc8c-bd89b6941cf4" containerName="cleanup" Jul 07 19:35:44.622879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.622835 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aab2161-05dc-4edd-bc8c-bd89b6941cf4" containerName="cleanup" Jul 07 19:35:44.625788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.625765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.628401 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.628372 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0e1e8b0-openshift-default-dockercfg-9clgr\"" Jul 07 19:35:44.646400 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.646359 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:44.661183 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.661140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:44.677615 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.677799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.677799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7fn\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.677799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.677799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677765 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.678064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.678064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.678064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.677978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.678064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.678006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.728082 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.727958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:44.779374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779335 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779751 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779751 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779751 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779907 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7fn\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779907 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779822 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.779907 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.779858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.780107 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.780042 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.780271 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.780243 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.780391 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.780290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.780391 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.780254 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.782385 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.782363 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.782587 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.782571 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.788949 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.788887 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7fn\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.789076 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.788884 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token\") pod \"e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:44.937434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:44.937390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:45.087643 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.087609 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:45.089997 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:35:45.089967 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48898f_cd81_477d_81cd_a2293273eb95.slice/crio-d35fb4ca349b899e4eccfcde9c0f22e89d21a086c5bded0b59e544a157b9f1ef WatchSource:0}: Error finding container d35fb4ca349b899e4eccfcde9c0f22e89d21a086c5bded0b59e544a157b9f1ef: Status 404 returned error can't find the container with id d35fb4ca349b899e4eccfcde9c0f22e89d21a086c5bded0b59e544a157b9f1ef Jul 07 19:35:45.092333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.092287 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:45.092421 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.092391 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:45.092462 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.092445 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:35:45.378470 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.378433 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" event={"ID":"7c48898f-cd81-477d-81cd-a2293273eb95","Type":"ContainerStarted","Data":"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8"} Jul 07 19:35:45.378704 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.378503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" event={"ID":"7c48898f-cd81-477d-81cd-a2293273eb95","Type":"ContainerStarted","Data":"d35fb4ca349b899e4eccfcde9c0f22e89d21a086c5bded0b59e544a157b9f1ef"} Jul 07 19:35:45.400747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.400689 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" podStartSLOduration=1.400671913 podStartE2EDuration="1.400671913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:45.398400826 +0000 UTC m=+1590.749049092" watchObservedRunningTime="2026-07-07 19:35:45.400671913 +0000 UTC m=+1590.751320149" Jul 07 19:35:45.883686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.883650 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:45.937778 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.937732 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:45.979962 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:45.979905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:46.101585 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:46.101545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:46.203183 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:46.203089 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:46.602038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:46.601950 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:46.699651 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:46.699619 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:46.943973 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:46.943941 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:47.390865 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:47.390837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:47.391754 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:47.391736 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:49.053659 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:49.053614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:53.854613 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:53.854576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:53.931040 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:53.931007 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:53.993906 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:53.993799 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:53.994114 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:53.994073 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" podUID="7c48898f-cd81-477d-81cd-a2293273eb95" containerName="istio-proxy" containerID="cri-o://52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8" gracePeriod=30 Jul 07 19:35:54.017050 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:54.017015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:54.112840 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:54.112686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:54.411409 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:54.411367 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:35:55.468789 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:55.468757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:35:57.059197 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.059128 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:57.207727 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207686 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207727 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207730 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207749 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207775 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207822 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207886 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h7fn\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207903 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.207991 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.207964 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.210165 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210135 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:57.210311 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210238 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert\") pod \"7c48898f-cd81-477d-81cd-a2293273eb95\" (UID: \"7c48898f-cd81-477d-81cd-a2293273eb95\") " Jul 07 19:35:57.210371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210314 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:57.210371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210330 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:57.210702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210636 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.210702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210664 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.210702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210679 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.210702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210695 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:35:57.210950 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.210733 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:57.212237 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.212212 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:57.212329 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.212280 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:35:57.212418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.212403 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:35:57.212591 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.212556 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn" (OuterVolumeSpecName: "kube-api-access-9h7fn") pod "7c48898f-cd81-477d-81cd-a2293273eb95" (UID: "7c48898f-cd81-477d-81cd-a2293273eb95"). InnerVolumeSpecName "kube-api-access-9h7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311469 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7fn\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-kube-api-access-9h7fn\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311502 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c48898f-cd81-477d-81cd-a2293273eb95-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311512 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311520 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c48898f-cd81-477d-81cd-a2293273eb95-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311528 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c48898f-cd81-477d-81cd-a2293273eb95-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.311565 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.311535 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c48898f-cd81-477d-81cd-a2293273eb95-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:35:57.457763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.457724 2568 generic.go:356] "Generic (PLEG): container finished" podID="7c48898f-cd81-477d-81cd-a2293273eb95" containerID="52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8" exitCode=0 Jul 07 19:35:57.457987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.457784 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" Jul 07 19:35:57.457987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.457801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" event={"ID":"7c48898f-cd81-477d-81cd-a2293273eb95","Type":"ContainerDied","Data":"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8"} Jul 07 19:35:57.457987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.457870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf" event={"ID":"7c48898f-cd81-477d-81cd-a2293273eb95","Type":"ContainerDied","Data":"d35fb4ca349b899e4eccfcde9c0f22e89d21a086c5bded0b59e544a157b9f1ef"} Jul 07 19:35:57.457987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.457898 2568 scope.go:117] "RemoveContainer" containerID="52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8" Jul 07 19:35:57.467886 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.467865 2568 scope.go:117] "RemoveContainer" containerID="52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8" Jul 07 19:35:57.468239 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:35:57.468215 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8\": container with ID starting with 52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8 not found: ID does not exist" containerID="52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8" Jul 07 19:35:57.468304 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.468249 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8"} err="failed to get container status \"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8\": rpc error: code = NotFound desc = could not find container \"52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8\": container with ID starting with 52bcfa2d3898d82d5fe7cb28a732a9c1901503c75f08cd10741ce9ed8dad4cd8 not found: ID does not exist" Jul 07 19:35:57.478127 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.478091 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:57.482585 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:57.482552 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0e1e8b0-openshift-default-568554fd9c-szcbf"] Jul 07 19:35:59.353021 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:35:59.352973 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c48898f-cd81-477d-81cd-a2293273eb95" path="/var/lib/kubelet/pods/7c48898f-cd81-477d-81cd-a2293273eb95/volumes" Jul 07 19:36:00.533646 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.533613 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:00.643089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.643050 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:00.772574 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.772537 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:00.773094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.773069 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c48898f-cd81-477d-81cd-a2293273eb95" containerName="istio-proxy" Jul 07 19:36:00.773094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.773093 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c48898f-cd81-477d-81cd-a2293273eb95" containerName="istio-proxy" Jul 07 19:36:00.773323 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.773169 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c48898f-cd81-477d-81cd-a2293273eb95" containerName="istio-proxy" Jul 07 19:36:00.776389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.776364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.778510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.778488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-810008b1-openshift-default-dockercfg-2g2x7\"" Jul 07 19:36:00.788632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.788603 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:00.818201 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.818164 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:00.916559 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.916525 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:00.948654 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.948821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.948821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.948821 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.948957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.948957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.949033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.948980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.949033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.949001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmbv\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:00.949095 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:00.949061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.006817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.006779 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:36:01.049677 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049677 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049677 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmbv\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.049980 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.049995 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050125 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050149 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050202 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050198 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050417 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050392 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.050735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.050716 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.052297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.052270 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.052574 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.052553 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.058459 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.058426 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.058595 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.058495 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmbv\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv\") pod \"e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.088394 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.088356 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:01.246933 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.246879 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:01.248715 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:36:01.248686 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e3263b_247d_4eef_acdd_54efbad950af.slice/crio-609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5 WatchSource:0}: Error finding container 609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5: Status 404 returned error can't find the container with id 609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5 Jul 07 19:36:01.250986 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.250951 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:01.251091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.251040 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:01.251091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.251084 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:01.483696 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.483656 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" event={"ID":"65e3263b-247d-4eef-acdd-54efbad950af","Type":"ContainerStarted","Data":"b58f04f5397a1a2e3aa2496e584848f151000d8e9f110a1802bf33385ee7a8d7"} Jul 07 19:36:01.483889 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.483722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" event={"ID":"65e3263b-247d-4eef-acdd-54efbad950af","Type":"ContainerStarted","Data":"609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5"} Jul 07 19:36:01.504832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.504758 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" podStartSLOduration=1.504735828 podStartE2EDuration="1.504735828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:01.504391537 +0000 UTC m=+1606.855039774" watchObservedRunningTime="2026-07-07 19:36:01.504735828 +0000 UTC m=+1606.855384068" Jul 07 19:36:01.949206 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:01.949174 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.045748 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.045716 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.089101 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.089061 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:02.094442 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.094404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:02.152686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.152647 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:02.156352 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.156325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.158560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.158533 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8pz6d\"" Jul 07 19:36:02.158716 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.158539 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-810008b1-serving-cert\"" Jul 07 19:36:02.167172 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.167121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.171976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.167591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.171976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.167770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmnl\" (UniqueName: \"kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.178974 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.178899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:02.196709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.196660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.268853 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.268752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.268853 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.268814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmnl\" (UniqueName: \"kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.269104 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.268862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.269104 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:36:02.268959 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-810008b1-serving-cert: secret "maas-api-e2e-ait-810008b1-serving-cert" not found Jul 07 19:36:02.269104 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:36:02.269038 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls podName:788fb5a6-2092-4fab-ba54-3b29f0a9cab8 nodeName:}" failed. No retries permitted until 2026-07-07 19:36:02.769015687 +0000 UTC m=+1608.119663905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls") pod "maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" (UID: "788fb5a6-2092-4fab-ba54-3b29f0a9cab8") : secret "maas-api-e2e-ait-810008b1-serving-cert" not found Jul 07 19:36:02.269582 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.269564 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.287668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.287628 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmnl\" (UniqueName: \"kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.336996 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.336946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.490013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.489982 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:02.491270 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.491251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:02.639279 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.639247 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.739561 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.739528 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:02.772878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.772845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.775853 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.775824 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") pod \"maas-api-e2e-ait-810008b1-67d6cb59b4-r67km\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.785418 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.785377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:02.939512 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.939484 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:02.941060 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:36:02.941031 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788fb5a6_2092_4fab_ba54_3b29f0a9cab8.slice/crio-3e23abbdab24364ec7a15d6cedfb2dd9d822b3b3f405e2bb9046ea2f8d832208 WatchSource:0}: Error finding container 3e23abbdab24364ec7a15d6cedfb2dd9d822b3b3f405e2bb9046ea2f8d832208: Status 404 returned error can't find the container with id 3e23abbdab24364ec7a15d6cedfb2dd9d822b3b3f405e2bb9046ea2f8d832208 Jul 07 19:36:02.942544 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:02.942525 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:36:03.496757 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:03.496721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" event={"ID":"788fb5a6-2092-4fab-ba54-3b29f0a9cab8","Type":"ContainerStarted","Data":"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f"} Jul 07 19:36:03.497285 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:03.496776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" event={"ID":"788fb5a6-2092-4fab-ba54-3b29f0a9cab8","Type":"ContainerStarted","Data":"3e23abbdab24364ec7a15d6cedfb2dd9d822b3b3f405e2bb9046ea2f8d832208"} Jul 07 19:36:03.497285 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:03.496805 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:03.516139 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:03.516084 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" podStartSLOduration=1.086795983 podStartE2EDuration="1.516063619s" totalImagesPullingTime="429.267636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:03.514620794 +0000 UTC m=+1608.865269057" watchObservedRunningTime="2026-07-07 19:36:03.516063619 +0000 UTC m=+1608.866711856" Jul 07 19:36:03.851652 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:03.851621 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:09.508432 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:09.508348 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:09.980717 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:09.980676 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:09.980983 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:09.980950 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" podUID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" containerName="maas-api" containerID="cri-o://9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f" gracePeriod=30 Jul 07 19:36:09.984511 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:09.984407 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:10.086107 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.086075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:10.264487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.264456 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:10.380338 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.380302 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca\") pod \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " Jul 07 19:36:10.380515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.380396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") pod \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " Jul 07 19:36:10.380566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.380518 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bmnl\" (UniqueName: \"kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl\") pod \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\" (UID: \"788fb5a6-2092-4fab-ba54-3b29f0a9cab8\") " Jul 07 19:36:10.380734 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.380706 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "788fb5a6-2092-4fab-ba54-3b29f0a9cab8" (UID: "788fb5a6-2092-4fab-ba54-3b29f0a9cab8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:36:10.380854 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.380837 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-openshift-service-ca\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:10.382531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.382499 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "788fb5a6-2092-4fab-ba54-3b29f0a9cab8" (UID: "788fb5a6-2092-4fab-ba54-3b29f0a9cab8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:36:10.382664 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.382587 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl" (OuterVolumeSpecName: "kube-api-access-6bmnl") pod "788fb5a6-2092-4fab-ba54-3b29f0a9cab8" (UID: "788fb5a6-2092-4fab-ba54-3b29f0a9cab8"). InnerVolumeSpecName "kube-api-access-6bmnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:10.481984 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.481951 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmnl\" (UniqueName: \"kubernetes.io/projected/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-kube-api-access-6bmnl\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:10.481984 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.481984 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788fb5a6-2092-4fab-ba54-3b29f0a9cab8-maas-api-tls\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:10.539868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.539762 2568 generic.go:356] "Generic (PLEG): container finished" podID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" containerID="9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f" exitCode=0 Jul 07 19:36:10.539868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.539812 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" Jul 07 19:36:10.539868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.539840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" event={"ID":"788fb5a6-2092-4fab-ba54-3b29f0a9cab8","Type":"ContainerDied","Data":"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f"} Jul 07 19:36:10.540464 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.539895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km" event={"ID":"788fb5a6-2092-4fab-ba54-3b29f0a9cab8","Type":"ContainerDied","Data":"3e23abbdab24364ec7a15d6cedfb2dd9d822b3b3f405e2bb9046ea2f8d832208"} Jul 07 19:36:10.540464 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.539933 2568 scope.go:117] "RemoveContainer" containerID="9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f" Jul 07 19:36:10.550998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.550972 2568 scope.go:117] "RemoveContainer" containerID="9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f" Jul 07 19:36:10.551331 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:36:10.551309 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f\": container with ID starting with 9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f not found: ID does not exist" containerID="9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f" Jul 07 19:36:10.551409 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.551344 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f"} err="failed to get container status \"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f\": rpc error: code = NotFound desc = could not find container \"9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f\": container with ID starting with 9315aae2d862520863544ed0c236f470e49262372328c972ef7185ef3f84da0f not found: ID does not exist" Jul 07 19:36:10.566806 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.566772 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:10.570899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.570862 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-810008b1-67d6cb59b4-r67km"] Jul 07 19:36:10.896318 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.896279 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:10.896522 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.896498 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" podUID="65e3263b-247d-4eef-acdd-54efbad950af" containerName="istio-proxy" containerID="cri-o://b58f04f5397a1a2e3aa2496e584848f151000d8e9f110a1802bf33385ee7a8d7" gracePeriod=30 Jul 07 19:36:10.927527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:10.927492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:11.014844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:11.014807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:11.356394 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:11.356301 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" path="/var/lib/kubelet/pods/788fb5a6-2092-4fab-ba54-3b29f0a9cab8/volumes" Jul 07 19:36:13.071284 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.071248 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:13.566681 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.566651 2568 generic.go:356] "Generic (PLEG): container finished" podID="65e3263b-247d-4eef-acdd-54efbad950af" containerID="b58f04f5397a1a2e3aa2496e584848f151000d8e9f110a1802bf33385ee7a8d7" exitCode=0 Jul 07 19:36:13.566877 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.566697 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" event={"ID":"65e3263b-247d-4eef-acdd-54efbad950af","Type":"ContainerDied","Data":"b58f04f5397a1a2e3aa2496e584848f151000d8e9f110a1802bf33385ee7a8d7"} Jul 07 19:36:13.566877 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.566745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" event={"ID":"65e3263b-247d-4eef-acdd-54efbad950af","Type":"ContainerDied","Data":"609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5"} Jul 07 19:36:13.566877 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.566768 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="609c02c49182ffa9e0e0147564befa3918aa2e0418e91163acef7dfc95575af5" Jul 07 19:36:13.570371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.570344 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:13.718151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718024 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718095 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718256 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718315 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718418 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718513 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718426 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:36:13.718574 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718500 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:13.718574 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718691 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718580 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdmbv\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718691 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718616 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718691 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718662 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket\") pod \"65e3263b-247d-4eef-acdd-54efbad950af\" (UID: \"65e3263b-247d-4eef-acdd-54efbad950af\") " Jul 07 19:36:13.718984 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.718948 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:13.719115 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.719093 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data" (OuterVolumeSpecName: "istio-data") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:13.719182 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.719109 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.719240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.719179 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:13.719240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.719192 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e3263b-247d-4eef-acdd-54efbad950af-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.719240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.719209 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.720690 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.720658 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:36:13.720800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.720703 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:13.720800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.720719 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token" (OuterVolumeSpecName: "istio-token") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:13.720800 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.720731 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv" (OuterVolumeSpecName: "kube-api-access-xdmbv") pod "65e3263b-247d-4eef-acdd-54efbad950af" (UID: "65e3263b-247d-4eef-acdd-54efbad950af"). InnerVolumeSpecName "kube-api-access-xdmbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:13.820120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820083 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.820120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820115 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e3263b-247d-4eef-acdd-54efbad950af-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.820120 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820125 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdmbv\" (UniqueName: \"kubernetes.io/projected/65e3263b-247d-4eef-acdd-54efbad950af-kube-api-access-xdmbv\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.820371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820134 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.820371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820143 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:13.820371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:13.820151 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e3263b-247d-4eef-acdd-54efbad950af-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:14.572950 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:14.572894 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb" Jul 07 19:36:14.596650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:14.596606 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:14.601148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:14.601117 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-810008b1-openshift-default-bc8d5d7d7-kdrkb"] Jul 07 19:36:15.354876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:15.354839 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e3263b-247d-4eef-acdd-54efbad950af" path="/var/lib/kubelet/pods/65e3263b-247d-4eef-acdd-54efbad950af/volumes" Jul 07 19:36:17.444223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.444192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:17.552281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.552245 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:17.639706 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.639667 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:17.640613 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640581 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e3263b-247d-4eef-acdd-54efbad950af" containerName="istio-proxy" Jul 07 19:36:17.640613 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640614 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e3263b-247d-4eef-acdd-54efbad950af" containerName="istio-proxy" Jul 07 19:36:17.640848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640639 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" containerName="maas-api" Jul 07 19:36:17.640848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640649 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" containerName="maas-api" Jul 07 19:36:17.640848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640760 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e3263b-247d-4eef-acdd-54efbad950af" containerName="istio-proxy" Jul 07 19:36:17.640848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.640776 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="788fb5a6-2092-4fab-ba54-3b29f0a9cab8" containerName="maas-api" Jul 07 19:36:17.645684 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.645658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.649414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.649388 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c5483003-openshift-default-dockercfg-9mswd\"" Jul 07 19:36:17.658510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.658473 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:17.680976 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.680944 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:17.757726 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.757726 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.757726 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqkd\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.758051 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.757972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.777849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.777821 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:17.859541 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqkd\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.859763 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.860133 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.860133 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.859826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.860133 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.860007 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.860698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.860667 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.860883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.860857 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.861193 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.861166 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.861396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.861370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.862560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.862529 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.870045 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.870003 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.871217 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.871192 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.871486 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.871463 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqkd\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd\") pod \"e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:17.958392 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:17.958347 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:18.112755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.112722 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:18.115153 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:36:18.115120 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cb6660_6147_43c0_8652_19d4be6ff982.slice/crio-ee99ddb15ee3481c23d7af93a32a6a807c56a9d8e4f5e7e6e734dfd72b21c7e1 WatchSource:0}: Error finding container ee99ddb15ee3481c23d7af93a32a6a807c56a9d8e4f5e7e6e734dfd72b21c7e1: Status 404 returned error can't find the container with id ee99ddb15ee3481c23d7af93a32a6a807c56a9d8e4f5e7e6e734dfd72b21c7e1 Jul 07 19:36:18.117443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.117409 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:18.117539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.117484 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:18.117539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.117513 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:36:18.599357 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.599324 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" event={"ID":"b9cb6660-6147-43c0-8652-19d4be6ff982","Type":"ContainerStarted","Data":"190a154ac0877e275a8cceb7bac8c7e80de7def024f83dbed67c61d05ce60146"} Jul 07 19:36:18.599765 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.599389 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" event={"ID":"b9cb6660-6147-43c0-8652-19d4be6ff982","Type":"ContainerStarted","Data":"ee99ddb15ee3481c23d7af93a32a6a807c56a9d8e4f5e7e6e734dfd72b21c7e1"} Jul 07 19:36:18.621414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.621359 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" podStartSLOduration=1.6213390890000001 podStartE2EDuration="1.621339089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:36:18.619655813 +0000 UTC m=+1623.970304051" watchObservedRunningTime="2026-07-07 19:36:18.621339089 +0000 UTC m=+1623.971987309" Jul 07 19:36:18.835240 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.835208 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:18.913899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.913864 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:18.958480 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.958444 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:18.987543 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:18.987512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:19.086323 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:19.086278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:19.419632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:19.419585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:19.519667 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:19.519634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:19.964439 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:19.964406 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:20.612385 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:20.612354 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:20.613500 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:20.613477 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:20.973034 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:20.972980 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:21.331284 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:21.331188 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:36:26.262306 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.261210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.356199 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.356160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.376295 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.376258 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:26.376499 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.376453 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" podUID="b9cb6660-6147-43c0-8652-19d4be6ff982" containerName="istio-proxy" containerID="cri-o://190a154ac0877e275a8cceb7bac8c7e80de7def024f83dbed67c61d05ce60146" gracePeriod=30 Jul 07 19:36:26.481687 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.481651 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:26.593065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:26.592944 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:27.018716 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:27.018686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:28.239200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:28.239155 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:29.676212 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.676172 2568 generic.go:356] "Generic (PLEG): container finished" podID="b9cb6660-6147-43c0-8652-19d4be6ff982" containerID="190a154ac0877e275a8cceb7bac8c7e80de7def024f83dbed67c61d05ce60146" exitCode=0 Jul 07 19:36:29.676639 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.676242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" event={"ID":"b9cb6660-6147-43c0-8652-19d4be6ff982","Type":"ContainerDied","Data":"190a154ac0877e275a8cceb7bac8c7e80de7def024f83dbed67c61d05ce60146"} Jul 07 19:36:29.757642 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.757609 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:29.788595 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.788812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788633 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.788812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788683 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.788812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788708 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.788812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788746 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.788812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.789148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788835 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.789148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788880 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.789148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788876 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:29.789148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.788965 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjqkd\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd\") pod \"b9cb6660-6147-43c0-8652-19d4be6ff982\" (UID: \"b9cb6660-6147-43c0-8652-19d4be6ff982\") " Jul 07 19:36:29.789148 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789072 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:29.789411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789293 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.789411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789358 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.789515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789403 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:29.789661 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789638 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:29.789824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.789798 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:36:29.791650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.791610 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:29.791782 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.791672 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd" (OuterVolumeSpecName: "kube-api-access-bjqkd") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "kube-api-access-bjqkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:36:29.791782 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.791707 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:36:29.791782 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.791705 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9cb6660-6147-43c0-8652-19d4be6ff982" (UID: "b9cb6660-6147-43c0-8652-19d4be6ff982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:36:29.889832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889791 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.889832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889826 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.889832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889834 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cb6660-6147-43c0-8652-19d4be6ff982-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.889832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889843 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.890143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889851 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cb6660-6147-43c0-8652-19d4be6ff982-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.890143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889859 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cb6660-6147-43c0-8652-19d4be6ff982-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:29.890143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:29.889868 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqkd\" (UniqueName: \"kubernetes.io/projected/b9cb6660-6147-43c0-8652-19d4be6ff982-kube-api-access-bjqkd\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:36:30.684531 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.684495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" event={"ID":"b9cb6660-6147-43c0-8652-19d4be6ff982","Type":"ContainerDied","Data":"ee99ddb15ee3481c23d7af93a32a6a807c56a9d8e4f5e7e6e734dfd72b21c7e1"} Jul 07 19:36:30.684964 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.684556 2568 scope.go:117] "RemoveContainer" containerID="190a154ac0877e275a8cceb7bac8c7e80de7def024f83dbed67c61d05ce60146" Jul 07 19:36:30.684964 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.684503 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc" Jul 07 19:36:30.701753 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.701713 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:36:30.715756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.715719 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:30.720530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:30.720495 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c5483003-openshift-default-856658cdcf-j6nkc"] Jul 07 19:36:31.354025 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:31.353988 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9cb6660-6147-43c0-8652-19d4be6ff982" path="/var/lib/kubelet/pods/b9cb6660-6147-43c0-8652-19d4be6ff982/volumes" Jul 07 19:36:33.450949 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:33.450890 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:33.551512 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:33.551483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:33.904151 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:33.904118 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:34.004676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:34.004643 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:34.104805 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:34.104770 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:35.440007 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:35.439974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:37.466654 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:37.466624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:40.801268 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:40.801226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:40.901701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:40.901669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:41.953064 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:41.953031 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:42.060868 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:42.060830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:42.153519 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:42.153490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:42.255992 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:42.255869 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:36:42.613656 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:36:42.613630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:08.581515 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:08.581477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:14.799913 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:14.799875 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:14.908117 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:14.908070 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:15.001065 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:15.001030 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:30.865807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:30.865769 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:37.041858 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:37.041777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:37.142787 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:37.142755 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:37.244156 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:37.244119 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:47.786106 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:47.786068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:49.380442 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:49.380411 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:54.885740 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:54.885709 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:54.987786 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:54.987757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:55.090386 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:55.090353 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:55.187474 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:55.187387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:55.992429 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:55.992400 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:56.092801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:56.092768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:57.741150 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:57.741116 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:37:58.309770 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:37:58.309724 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:38:04.005089 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:04.005033 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:04.074185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:04.074152 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:04.695560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:04.695522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:07.213699 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:07.213659 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:38:11.146888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.146851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:11.211100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.211064 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:11.384547 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.384517 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:11.676613 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.676572 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:11.677099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.677083 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9cb6660-6147-43c0-8652-19d4be6ff982" containerName="istio-proxy" Jul 07 19:38:11.677099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.677100 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9cb6660-6147-43c0-8652-19d4be6ff982" containerName="istio-proxy" Jul 07 19:38:11.677222 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.677187 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9cb6660-6147-43c0-8652-19d4be6ff982" containerName="istio-proxy" Jul 07 19:38:11.680577 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.680555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.683022 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.682999 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67a065b1-openshift-default-dockercfg-xsn6f\"" Jul 07 19:38:11.700207 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.700161 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:11.702629 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.702586 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:11.785773 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.785773 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785903 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mrp\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.785981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.786003 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.786040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.786209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.786069 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.886832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.886787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mrp\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.886832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.886839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.886863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.886902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.886973 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887129 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887493 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887351 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887493 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887686 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887832 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887728 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.887885 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.887824 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.889972 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.889943 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.890130 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.890112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.895387 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.895354 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.895549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.895530 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mrp\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp\") pod \"e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:11.991910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:11.991806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:12.106072 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.106037 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:12.144649 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.144608 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:12.148537 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:38:12.148503 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158edc47_95ce_44a5_81d1_91559c989871.slice/crio-971de601283a71144d93b5325a9d7c72878d32d9892636f6e70b9737cf57f80b WatchSource:0}: Error finding container 971de601283a71144d93b5325a9d7c72878d32d9892636f6e70b9737cf57f80b: Status 404 returned error can't find the container with id 971de601283a71144d93b5325a9d7c72878d32d9892636f6e70b9737cf57f80b Jul 07 19:38:12.151225 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.151189 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:12.151349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.151279 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:12.151349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.151322 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:12.298788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.298699 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerStarted","Data":"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464"} Jul 07 19:38:12.298788 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.298761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerStarted","Data":"971de601283a71144d93b5325a9d7c72878d32d9892636f6e70b9737cf57f80b"} Jul 07 19:38:12.320691 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.320631 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podStartSLOduration=1.320612673 podStartE2EDuration="1.320612673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:38:12.318161403 +0000 UTC m=+1737.668809641" watchObservedRunningTime="2026-07-07 19:38:12.320612673 +0000 UTC m=+1737.671260909" Jul 07 19:38:12.992718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:12.992678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:13.997319 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:13.997275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:14.310611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:14.310519 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:14.311499 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:14.311480 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:14.512545 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:14.512509 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:14.621057 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:14.621020 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:14.713443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:14.713413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:15.187906 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:15.187866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:15.286468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:15.286434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:18.704807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:18.704764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:38:19.000461 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:19.000366 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:19.638801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:19.638760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:19.742678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:19.742635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:20.063439 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:20.063340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:26.716374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:26.716290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:26.782667 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:26.782572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:27.150859 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.150832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:27.398792 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.398758 2568 generic.go:356] "Generic (PLEG): container finished" podID="158edc47-95ce-44a5-81d1-91559c989871" containerID="20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464" exitCode=0 Jul 07 19:38:27.399007 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.398830 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerDied","Data":"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464"} Jul 07 19:38:27.399345 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.399328 2568 scope.go:117] "RemoveContainer" containerID="20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464" Jul 07 19:38:27.400498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.400473 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:27.400576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.400562 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:27.400612 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.400603 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:27.449682 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.449641 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:27.550173 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:27.550127 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:28.407416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:28.407381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerStarted","Data":"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf"} Jul 07 19:38:28.992181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:28.992141 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:28.993563 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:28.993534 2568 patch_prober.go:28] interesting pod/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 19:38:28.993680 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:28.993587 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 19:38:29.310824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:29.310722 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:29.992564 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:29.992525 2568 patch_prober.go:28] interesting pod/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 19:38:29.992975 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:29.992587 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 19:38:30.705676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:30.705640 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:38:30.992628 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:30.992537 2568 patch_prober.go:28] interesting pod/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 19:38:30.992628 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:30.992594 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 19:38:31.993018 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:31.992982 2568 patch_prober.go:28] interesting pod/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 19:38:31.993480 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:31.993340 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 19:38:32.996987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:32.996947 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:32.998266 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:32.998248 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:33.932374 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:33.932344 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.032693 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.032661 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.176175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.176134 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:34.192268 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.192186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.444858 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.444753 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" containerID="cri-o://1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf" gracePeriod=30 Jul 07 19:38:34.465216 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.465182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.592430 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.592394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.682929 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.682884 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:34.803797 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:34.803718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:39.722961 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.722873 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:39.780378 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780342 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780422 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780476 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780573 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780603 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780599 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780621 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6mrp\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780650 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780694 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token\") pod \"158edc47-95ce-44a5-81d1-91559c989871\" (UID: \"158edc47-95ce-44a5-81d1-91559c989871\") " Jul 07 19:38:39.780879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780725 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:39.780879 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.780776 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:39.782044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.781218 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.782044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.781248 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.782044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.781495 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:38:39.782044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.781846 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:39.782355 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.782066 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data" (OuterVolumeSpecName: "istio-data") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:39.783262 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.783236 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:39.783400 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.783311 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token" (OuterVolumeSpecName: "istio-token") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:39.783548 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.783522 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp" (OuterVolumeSpecName: "kube-api-access-d6mrp") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "kube-api-access-d6mrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:39.783906 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.783881 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "158edc47-95ce-44a5-81d1-91559c989871" (UID: "158edc47-95ce-44a5-81d1-91559c989871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:38:39.882517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882476 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882513 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882517 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882525 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158edc47-95ce-44a5-81d1-91559c989871-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882533 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882542 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158edc47-95ce-44a5-81d1-91559c989871-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882551 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mrp\" (UniqueName: \"kubernetes.io/projected/158edc47-95ce-44a5-81d1-91559c989871-kube-api-access-d6mrp\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:39.882771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:39.882559 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158edc47-95ce-44a5-81d1-91559c989871-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:38:40.482807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.482772 2568 generic.go:356] "Generic (PLEG): container finished" podID="158edc47-95ce-44a5-81d1-91559c989871" containerID="1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf" exitCode=0 Jul 07 19:38:40.483010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.482822 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" Jul 07 19:38:40.483010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.482847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerDied","Data":"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf"} Jul 07 19:38:40.483010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.482899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd" event={"ID":"158edc47-95ce-44a5-81d1-91559c989871","Type":"ContainerDied","Data":"971de601283a71144d93b5325a9d7c72878d32d9892636f6e70b9737cf57f80b"} Jul 07 19:38:40.483010 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.482935 2568 scope.go:117] "RemoveContainer" containerID="1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf" Jul 07 19:38:40.493549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.493527 2568 scope.go:117] "RemoveContainer" containerID="20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464" Jul 07 19:38:40.503789 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.503765 2568 scope.go:117] "RemoveContainer" containerID="1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf" Jul 07 19:38:40.504255 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:38:40.504225 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf\": container with ID starting with 1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf not found: ID does not exist" containerID="1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf" Jul 07 19:38:40.504390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.504263 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf"} err="failed to get container status \"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf\": rpc error: code = NotFound desc = could not find container \"1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf\": container with ID starting with 1a64109607909a609abe9612261bf013d44e808e7d7b9f2ac58661a24cad8abf not found: ID does not exist" Jul 07 19:38:40.504390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.504283 2568 scope.go:117] "RemoveContainer" containerID="20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464" Jul 07 19:38:40.504662 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:38:40.504636 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464\": container with ID starting with 20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464 not found: ID does not exist" containerID="20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464" Jul 07 19:38:40.504708 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.504671 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464"} err="failed to get container status \"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464\": rpc error: code = NotFound desc = could not find container \"20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464\": container with ID starting with 20d15f317bb0426acd0dbf2316dc46560fcb624c47ec84f7d1eaf4089a95e464 not found: ID does not exist" Jul 07 19:38:40.507137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.507114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:40.513044 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:40.513016 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67a065b1-openshift-default-6968b8f755-x7lxd"] Jul 07 19:38:41.354071 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:41.354041 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158edc47-95ce-44a5-81d1-91559c989871" path="/var/lib/kubelet/pods/158edc47-95ce-44a5-81d1-91559c989871/volumes" Jul 07 19:38:41.470558 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:41.470523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:41.585350 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:41.585296 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:41.672017 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:41.671984 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:41.808862 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:41.808828 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:42.368786 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:42.368749 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:42.471067 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:42.471033 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:48.320530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:48.320496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:48.418793 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:48.418759 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:54.998068 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:54.998029 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.271640 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.271549 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:38:55.272136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.272109 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:38:55.272136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.272131 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:38:55.272252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.272145 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:38:55.272252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.272151 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:38:55.272252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.272220 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:38:55.275431 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.275408 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.277888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.277862 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6fe61d50-openshift-default-dockercfg-mrncf\"" Jul 07 19:38:55.289964 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.289933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:38:55.303551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.303516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.402302 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.402274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:55.445718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.445674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.445958 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.445735 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446103 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cnx\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446397 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446262 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446397 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.446510 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.446401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cnx\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547628 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.547848 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.547655 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.548060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.548030 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.548203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.548038 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.548203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.548071 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.549736 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.549712 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.549987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.549970 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.557871 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.557838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cnx\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.562185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.562157 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token\") pod \"e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.586586 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.586547 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:55.745647 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.745616 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:38:55.746898 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:38:55.746873 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9d00d7_1524_4493_a563_4fe52ffd0488.slice/crio-a736a169f62860b0c4a85b13424f0818854b122d8e4ce2e88ff9f859d7806b41 WatchSource:0}: Error finding container a736a169f62860b0c4a85b13424f0818854b122d8e4ce2e88ff9f859d7806b41: Status 404 returned error can't find the container with id a736a169f62860b0c4a85b13424f0818854b122d8e4ce2e88ff9f859d7806b41 Jul 07 19:38:55.749127 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.749093 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:55.749192 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.749166 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:55.749226 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:55.749197 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:38:56.581032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:56.580992 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerStarted","Data":"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360"} Jul 07 19:38:56.581463 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:56.581045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerStarted","Data":"a736a169f62860b0c4a85b13424f0818854b122d8e4ce2e88ff9f859d7806b41"} Jul 07 19:38:56.587169 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:56.587139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:56.592488 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:56.592460 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:56.604219 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:56.604167 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podStartSLOduration=1.604147696 podStartE2EDuration="1.604147696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:38:56.601629202 +0000 UTC m=+1781.952277515" watchObservedRunningTime="2026-07-07 19:38:56.604147696 +0000 UTC m=+1781.954795934" Jul 07 19:38:57.586947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:57.586882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:57.588030 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:57.588011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:38:58.506702 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:58.506667 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:58.631591 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:58.631552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:58.751883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:58.751847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:58.847969 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:58.847840 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:59.303108 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:59.303074 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:38:59.400431 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:38:59.400398 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:03.001056 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:03.001022 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:04.006662 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:04.006624 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:39:12.588272 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.588238 2568 patch_prober.go:28] interesting pod/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 19:39:12.588664 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.588308 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 19:39:12.697741 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.697708 2568 generic.go:356] "Generic (PLEG): container finished" podID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerID="38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360" exitCode=0 Jul 07 19:39:12.697935 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.697764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerDied","Data":"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360"} Jul 07 19:39:12.698243 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.698224 2568 scope.go:117] "RemoveContainer" containerID="38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360" Jul 07 19:39:12.699239 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.699211 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:39:12.699298 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.699281 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:39:12.699335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:12.699312 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:39:13.705456 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:13.705424 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerStarted","Data":"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74"} Jul 07 19:39:14.587242 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:14.587201 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:39:14.587539 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:14.587516 2568 patch_prober.go:28] interesting pod/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 19:39:14.587604 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:14.587570 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 19:39:15.588075 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:15.588033 2568 patch_prober.go:28] interesting pod/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 19:39:15.588478 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:15.588101 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 19:39:16.587650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:16.587614 2568 patch_prober.go:28] interesting pod/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 19:39:16.587853 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:16.587687 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 19:39:17.587057 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:17.587016 2568 patch_prober.go:28] interesting pod/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 19:39:17.587507 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:17.587074 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 19:39:18.592032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:18.592001 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:39:18.592450 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:18.592338 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:39:18.593251 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:18.593234 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:39:20.645473 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:20.645432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:23.917046 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:23.917010 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:39:39.812057 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:39.812018 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:39:40.640031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:40.639996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:40.916518 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:40.916432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:46.646038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:46.646006 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:51.729028 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:51.728992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:39:52.106505 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:39:52.106412 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:40:29.444268 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:29.444235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:29.985116 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:29.985055 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:32.608963 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:32.608928 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:40:36.948798 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:36.948705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:37.034698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:37.034663 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:37.236453 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:37.236359 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:40:37.236670 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:37.236635 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" containerID="cri-o://08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74" gracePeriod=30 Jul 07 19:40:37.290909 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:37.290877 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:37.497493 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:37.497415 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:38.546388 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:38.546355 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:38.646861 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:38.646830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:39.431733 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:39.431692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:39.536812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:39.536764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:39.637955 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:39.637903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:40.978144 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:40.978114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:41.629951 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:41.629883 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:40:42.510136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.510061 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:40:42.553465 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553428 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553465 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553466 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553498 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553539 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553581 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553619 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.553729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553695 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2cnx\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.554012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553737 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.554012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553769 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo\") pod \"7d9d00d7-1524-4493-a563-4fe52ffd0488\" (UID: \"7d9d00d7-1524-4493-a563-4fe52ffd0488\") " Jul 07 19:40:42.554012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553804 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:42.554012 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.553950 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:40:42.554204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554044 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:42.554204 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554185 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.554300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554203 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9d00d7-1524-4493-a563-4fe52ffd0488-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.554300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554217 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.554300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554246 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:42.554586 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.554557 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:42.556102 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.556070 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx" (OuterVolumeSpecName: "kube-api-access-x2cnx") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "kube-api-access-x2cnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:40:42.556367 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.556347 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:40:42.556445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.556410 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:40:42.556508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.556456 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d9d00d7-1524-4493-a563-4fe52ffd0488" (UID: "7d9d00d7-1524-4493-a563-4fe52ffd0488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:42.655099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655056 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.655099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655095 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.655099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655107 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cnx\" (UniqueName: \"kubernetes.io/projected/7d9d00d7-1524-4493-a563-4fe52ffd0488-kube-api-access-x2cnx\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.655360 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655122 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.655360 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655133 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:42.655360 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:42.655144 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9d00d7-1524-4493-a563-4fe52ffd0488-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:40:43.240736 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.240696 2568 generic.go:356] "Generic (PLEG): container finished" podID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerID="08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74" exitCode=0 Jul 07 19:40:43.240934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.240750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerDied","Data":"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74"} Jul 07 19:40:43.240934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.240759 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" Jul 07 19:40:43.240934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.240801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg" event={"ID":"7d9d00d7-1524-4493-a563-4fe52ffd0488","Type":"ContainerDied","Data":"a736a169f62860b0c4a85b13424f0818854b122d8e4ce2e88ff9f859d7806b41"} Jul 07 19:40:43.240934 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.240829 2568 scope.go:117] "RemoveContainer" containerID="08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74" Jul 07 19:40:43.251199 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.251169 2568 scope.go:117] "RemoveContainer" containerID="38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360" Jul 07 19:40:43.262352 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.262311 2568 scope.go:117] "RemoveContainer" containerID="08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74" Jul 07 19:40:43.262718 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:40:43.262697 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74\": container with ID starting with 08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74 not found: ID does not exist" containerID="08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74" Jul 07 19:40:43.262769 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.262728 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74"} err="failed to get container status \"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74\": rpc error: code = NotFound desc = could not find container \"08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74\": container with ID starting with 08181b33445951bf19b570b0d1b1d66d3756992fd11cac08ec27d53108c6cb74 not found: ID does not exist" Jul 07 19:40:43.262769 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.262747 2568 scope.go:117] "RemoveContainer" containerID="38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360" Jul 07 19:40:43.263073 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:40:43.263055 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360\": container with ID starting with 38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360 not found: ID does not exist" containerID="38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360" Jul 07 19:40:43.263127 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.263079 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360"} err="failed to get container status \"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360\": rpc error: code = NotFound desc = could not find container \"38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360\": container with ID starting with 38308b64641082932817da4ef63cbdd4dae23e2420bfaa7f0f20a77528014360 not found: ID does not exist" Jul 07 19:40:43.270039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.270006 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:40:43.276697 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.276662 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6fe61d50-openshift-default-6b9974cf47-g54bg"] Jul 07 19:40:43.355605 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:43.355564 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" path="/var/lib/kubelet/pods/7d9d00d7-1524-4493-a563-4fe52ffd0488/volumes" Jul 07 19:40:48.494742 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:48.494682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:48.607579 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:48.607549 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:49.206070 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:49.206034 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:53.307502 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:53.307464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:40:56.381837 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:56.381802 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:56.533180 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:56.533135 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:40:56.712525 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:40:56.712448 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:07.128107 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:07.128069 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:07.484300 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:07.484203 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:28.478709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:28.478663 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:30.373061 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:30.373025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:35.679296 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:35.679261 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:35.778566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:35.778533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:36.043052 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.042962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:36.361191 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361158 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:41:36.361774 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361758 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:36.361820 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361777 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:36.361898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361887 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:36.361957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361903 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:36.361957 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.361928 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="158edc47-95ce-44a5-81d1-91559c989871" containerName="istio-proxy" Jul 07 19:41:36.364551 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.364529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.366735 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.366714 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1aa2d9-openshift-default-dockercfg-dcvxk\"" Jul 07 19:41:36.380860 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.380831 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:41:36.408067 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.408036 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:36.477477 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477995 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhkc\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477995 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.477995 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.477854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.578866 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.578826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.578887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.578949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.578977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579099 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579345 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579402 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579461 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhkc\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579461 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579418 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579487 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579507 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.579893 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.579869 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.581615 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.581594 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.581898 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.581881 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.588686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.588651 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.589070 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.589049 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhkc\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc\") pod \"e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.669332 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.669230 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:36.675114 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.675090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:36.834067 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.834034 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:41:36.835810 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:41:36.835777 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59780ee4_56f3_421d_ab6f_421a2d9350ad.slice/crio-9c13b63b1efeecaaee6c8ef4a21b2b834cd6234aab9c912eb389828e6f269e6c WatchSource:0}: Error finding container 9c13b63b1efeecaaee6c8ef4a21b2b834cd6234aab9c912eb389828e6f269e6c: Status 404 returned error can't find the container with id 9c13b63b1efeecaaee6c8ef4a21b2b834cd6234aab9c912eb389828e6f269e6c Jul 07 19:41:36.838187 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.838154 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:36.838281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.838222 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:36.838281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:36.838249 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:37.552234 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:37.552196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerStarted","Data":"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256"} Jul 07 19:41:37.552421 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:37.552250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerStarted","Data":"9c13b63b1efeecaaee6c8ef4a21b2b834cd6234aab9c912eb389828e6f269e6c"} Jul 07 19:41:37.594254 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:37.594200 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podStartSLOduration=1.5941834849999998 podStartE2EDuration="1.594183485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:41:37.592891679 +0000 UTC m=+1942.943539941" watchObservedRunningTime="2026-07-07 19:41:37.594183485 +0000 UTC m=+1942.944831721" Jul 07 19:41:37.675416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:37.675382 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:37.680676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:37.680647 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:38.557888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:38.557855 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:38.559039 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:38.559014 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:39.609355 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:39.609312 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:39.867482 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:39.867395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:40.157091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:40.157002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:40.711408 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:40.711364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:40.877730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:40.877699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:42.531054 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:42.531016 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:42.737777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:42.737742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:44.141795 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.141753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:44.464198 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.464106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:44.714888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.714794 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:41:44.718936 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.716332 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:44.718936 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.716368 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9d00d7-1524-4493-a563-4fe52ffd0488" containerName="istio-proxy" Jul 07 19:41:44.721000 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.719965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.723143 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.723109 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c47ca8-openshift-default-dockercfg-7tpqk\"" Jul 07 19:41:44.745104 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.745058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:41:44.864312 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtpr\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864484 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.864799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.864672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:44.965698 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.965856 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965754 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.965856 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.965856 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.965856 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965861 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtpr\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.965974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966100 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966291 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966096 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966291 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966124 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966291 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966223 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966320 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.966815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.966792 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.968142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.968118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.968349 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.968331 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.975278 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.975248 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.975455 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.975430 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtpr\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr\") pod \"e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:44.990310 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:44.990276 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:45.036019 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.035980 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:45.199038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.199005 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:41:45.200963 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:41:45.200932 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded790476_9093_4deb_b493_5502ab395b5d.slice/crio-95516126b63a98521818c285b1bd1cabc0d7ca00a5e2aa2a337829a946f555b2 WatchSource:0}: Error finding container 95516126b63a98521818c285b1bd1cabc0d7ca00a5e2aa2a337829a946f555b2: Status 404 returned error can't find the container with id 95516126b63a98521818c285b1bd1cabc0d7ca00a5e2aa2a337829a946f555b2 Jul 07 19:41:45.203156 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.203121 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:45.203281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.203205 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:45.203281 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.203247 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:45.601232 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.601129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerStarted","Data":"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064"} Jul 07 19:41:45.601232 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.601185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerStarted","Data":"95516126b63a98521818c285b1bd1cabc0d7ca00a5e2aa2a337829a946f555b2"} Jul 07 19:41:45.647209 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:45.647155 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podStartSLOduration=1.647135856 podStartE2EDuration="1.647135856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:41:45.646394219 +0000 UTC m=+1950.997042459" watchObservedRunningTime="2026-07-07 19:41:45.647135856 +0000 UTC m=+1950.997784093" Jul 07 19:41:46.036462 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:46.036423 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:46.041540 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:46.041511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:46.607594 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:46.607561 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:46.608668 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:46.608645 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:41:47.043995 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:47.043954 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:41:47.705334 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:47.705290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:47.986644 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:47.986549 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.248790 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:48.248693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.722398 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:48.722358 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:48.942205 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:48.942145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:52.167768 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:52.167738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:41:54.600162 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:54.600110 2568 patch_prober.go:28] interesting pod/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 19:41:54.600660 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:54.600187 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 19:41:55.670669 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.670634 2568 generic.go:356] "Generic (PLEG): container finished" podID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerID="f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256" exitCode=0 Jul 07 19:41:55.671142 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.670709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerDied","Data":"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256"} Jul 07 19:41:55.671257 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.671239 2568 scope.go:117] "RemoveContainer" containerID="f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256" Jul 07 19:41:55.672363 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.672331 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:55.672475 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.672401 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:55.672475 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:55.672455 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:41:56.684569 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:56.684527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerStarted","Data":"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9"} Jul 07 19:41:57.221340 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:57.221294 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:41:57.675576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:57.675531 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:41:57.675807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:57.675675 2568 patch_prober.go:28] interesting pod/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 19:41:57.675807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:57.675724 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 19:41:58.676306 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:58.676270 2568 patch_prober.go:28] interesting pod/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 19:41:58.676733 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:58.676327 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 19:41:59.676547 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:59.676509 2568 patch_prober.go:28] interesting pod/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 19:41:59.677091 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:41:59.676579 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 19:42:00.676035 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:00.675981 2568 patch_prober.go:28] interesting pod/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 19:42:00.676250 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:00.676057 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 19:42:01.609575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.609544 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:01.610006 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.609619 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:01.679054 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.678976 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:42:01.679282 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.679256 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:42:01.680181 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.680163 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:42:01.721193 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.721159 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed790476-9093-4deb-b493-5502ab395b5d" containerID="94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064" exitCode=0 Jul 07 19:42:01.721389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.721227 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerDied","Data":"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064"} Jul 07 19:42:01.721732 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.721719 2568 scope.go:117] "RemoveContainer" containerID="94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064" Jul 07 19:42:01.722730 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.722702 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:42:01.722824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.722781 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:42:01.722824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.722819 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 07 19:42:01.728700 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.728679 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:01.883201 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:01.883167 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:02.730537 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:02.730504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerStarted","Data":"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede"} Jul 07 19:42:03.037019 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:03.036873 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:42:03.037395 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:03.037361 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:03.037488 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:03.037429 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:04.036817 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:04.036771 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:04.037532 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:04.037497 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:05.037054 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:05.037010 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:05.037602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:05.037079 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:06.037203 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:06.037154 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:06.037742 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:06.037246 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:07.037231 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:07.037190 2568 patch_prober.go:28] interesting pod/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 19:42:07.037873 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:07.037837 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 19:42:08.041017 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:08.040988 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:42:08.041552 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:08.041528 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:42:08.042441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:08.042421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:42:09.655443 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:09.655406 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:09.908571 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:09.908468 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:10.160033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:10.159939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:13.213566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:13.213526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:42:15.445987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:15.445948 2568 scope.go:117] "RemoveContainer" containerID="b58f04f5397a1a2e3aa2496e584848f151000d8e9f110a1802bf33385ee7a8d7" Jul 07 19:42:23.513403 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:23.513363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:42:26.635602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:26.635557 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:33.118299 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:33.118250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:42:44.896212 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:44.896177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:45.286679 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:45.286592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:45.390778 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:45.390737 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:51.205386 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:51.205338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:52.411760 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:52.411724 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:42:55.701816 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:42:55.701781 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:02.888988 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:02.888953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:06.415411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:06.415369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:43:17.358576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:17.358541 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:17.731285 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:17.731250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:17.828050 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:17.828017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:22.794849 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:22.794817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:25.103576 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:25.103537 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:28.800191 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:28.800152 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:43:43.245140 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:43.245054 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:43.341317 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:43.341290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:53.091178 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:53.091127 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:43:58.935212 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:43:58.935180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:00.535171 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:00.535136 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:04.313283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:04.313242 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:44:06.819060 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:06.819021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:06.924596 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:06.924563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:07.138743 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:07.138710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:07.373595 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:07.373561 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:08.327390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:08.327356 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:09.319140 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:09.319109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:09.605226 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:09.605143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:09.963013 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:09.962982 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:10.145974 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:10.145943 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:10.266175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:10.266077 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:10.463857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:10.463824 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:15.137430 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:15.137394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:18.214807 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:18.214760 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:44:52.482210 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:52.482177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:52.754527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:52.754428 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:44:52.754713 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:52.754665 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" containerID="cri-o://3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9" gracePeriod=30 Jul 07 19:44:52.785336 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:52.785289 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.009097 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.008971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.189796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.189758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.579964 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.579933 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.722001 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.721912 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.824217 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.824128 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:53.985878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:53.985846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:44:56.506110 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:56.506058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:44:58.023771 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.023685 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:44:58.129609 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129566 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.129818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129641 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.129818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129671 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.129818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.129818 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129735 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.130066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129830 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.130066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129858 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfhkc\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.130066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129887 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.130066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.129975 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert\") pod \"59780ee4-56f3-421d-ab6f-421a2d9350ad\" (UID: \"59780ee4-56f3-421d-ab6f-421a2d9350ad\") " Jul 07 19:44:58.130066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130018 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:44:58.130314 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130087 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:44:58.130314 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130126 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:44:58.130434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130392 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.130434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130413 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.130434 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130428 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.130556 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130454 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:44:58.130556 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.130481 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:44:58.132166 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.132141 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:44:58.132283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.132197 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc" (OuterVolumeSpecName: "kube-api-access-nfhkc") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "kube-api-access-nfhkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:44:58.132396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.132378 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:44:58.132396 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.132386 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "59780ee4-56f3-421d-ab6f-421a2d9350ad" (UID: "59780ee4-56f3-421d-ab6f-421a2d9350ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:44:58.231526 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231482 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.231526 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231517 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.231526 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231526 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.231526 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231537 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhkc\" (UniqueName: \"kubernetes.io/projected/59780ee4-56f3-421d-ab6f-421a2d9350ad-kube-api-access-nfhkc\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.231815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231547 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59780ee4-56f3-421d-ab6f-421a2d9350ad-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.231815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.231556 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59780ee4-56f3-421d-ab6f-421a2d9350ad-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:44:58.801674 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.801639 2568 generic.go:356] "Generic (PLEG): container finished" podID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerID="3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9" exitCode=0 Jul 07 19:44:58.801884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.801688 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" Jul 07 19:44:58.801884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.801720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerDied","Data":"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9"} Jul 07 19:44:58.801884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.801775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf" event={"ID":"59780ee4-56f3-421d-ab6f-421a2d9350ad","Type":"ContainerDied","Data":"9c13b63b1efeecaaee6c8ef4a21b2b834cd6234aab9c912eb389828e6f269e6c"} Jul 07 19:44:58.801884 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.801798 2568 scope.go:117] "RemoveContainer" containerID="3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9" Jul 07 19:44:58.812845 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.812823 2568 scope.go:117] "RemoveContainer" containerID="f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256" Jul 07 19:44:58.822508 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.822483 2568 scope.go:117] "RemoveContainer" containerID="3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9" Jul 07 19:44:58.822898 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:44:58.822853 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9\": container with ID starting with 3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9 not found: ID does not exist" containerID="3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9" Jul 07 19:44:58.823031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.822907 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9"} err="failed to get container status \"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9\": rpc error: code = NotFound desc = could not find container \"3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9\": container with ID starting with 3373d99045a71ffad7cf07e030b982d5d9ee29c4eeab06407eb78d03e732f3d9 not found: ID does not exist" Jul 07 19:44:58.823031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.822970 2568 scope.go:117] "RemoveContainer" containerID="f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256" Jul 07 19:44:58.823380 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:44:58.823357 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256\": container with ID starting with f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256 not found: ID does not exist" containerID="f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256" Jul 07 19:44:58.823474 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.823390 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256"} err="failed to get container status \"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256\": rpc error: code = NotFound desc = could not find container \"f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256\": container with ID starting with f2deedfd650a98dea0369ebca0bf2c06e25940c239517c9d5b5a14a4d68ef256 not found: ID does not exist" Jul 07 19:44:58.826158 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.826133 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:44:58.830707 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:58.830670 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa2d9-openshift-default-75d569984f-8jwkf"] Jul 07 19:44:59.353351 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:44:59.353315 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" path="/var/lib/kubelet/pods/59780ee4-56f3-421d-ab6f-421a2d9350ad/volumes" Jul 07 19:45:00.104610 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.104581 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:00.208831 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.208796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:00.352447 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.352415 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:45:00.352687 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.352655 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" containerID="cri-o://31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede" gracePeriod=30 Jul 07 19:45:00.398801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.398767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:00.596824 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:00.596793 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:01.037003 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:01.036967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:01.310137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:01.310054 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:01.660862 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:01.660822 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:03.828265 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:03.828229 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:03.946566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:03.946532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:04.059387 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:04.059357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:04.399498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:04.399466 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:04.536876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:04.536841 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:05.625894 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.625818 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:45:05.700878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.700842 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.700878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.700881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.700901 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.700944 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.700977 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701007 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701084 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.701411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701236 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:45:05.701411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701271 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:45:05.701411 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701284 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:45:05.701584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701430 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-credential-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.701584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701445 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-socket\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.701584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.701456 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-workload-certs\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.702201 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.702177 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:45:05.703214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.703194 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:45:05.703315 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.703295 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:05.703370 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.703307 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:45:05.801876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.801833 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.801876 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.801881 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgtpr\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr\") pod \"ed790476-9093-4deb-b493-5502ab395b5d\" (UID: \"ed790476-9093-4deb-b493-5502ab395b5d\") " Jul 07 19:45:05.802141 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.802062 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-data\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.802141 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.802074 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed790476-9093-4deb-b493-5502ab395b5d-istio-envoy\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.802141 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.802082 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-istio-token\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.802141 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.802091 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed790476-9093-4deb-b493-5502ab395b5d-istio-podinfo\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.802297 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.802242 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:45:05.804029 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.804001 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr" (OuterVolumeSpecName: "kube-api-access-kgtpr") pod "ed790476-9093-4deb-b493-5502ab395b5d" (UID: "ed790476-9093-4deb-b493-5502ab395b5d"). InnerVolumeSpecName "kube-api-access-kgtpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:05.846687 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.846644 2568 generic.go:356] "Generic (PLEG): container finished" podID="ed790476-9093-4deb-b493-5502ab395b5d" containerID="31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede" exitCode=0 Jul 07 19:45:05.846888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.846693 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" Jul 07 19:45:05.846888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.846724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerDied","Data":"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede"} Jul 07 19:45:05.846888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.846778 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5" event={"ID":"ed790476-9093-4deb-b493-5502ab395b5d","Type":"ContainerDied","Data":"95516126b63a98521818c285b1bd1cabc0d7ca00a5e2aa2a337829a946f555b2"} Jul 07 19:45:05.846888 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.846800 2568 scope.go:117] "RemoveContainer" containerID="31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede" Jul 07 19:45:05.857555 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.857534 2568 scope.go:117] "RemoveContainer" containerID="94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064" Jul 07 19:45:05.867632 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.867607 2568 scope.go:117] "RemoveContainer" containerID="31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede" Jul 07 19:45:05.867996 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:05.867967 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede\": container with ID starting with 31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede not found: ID does not exist" containerID="31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede" Jul 07 19:45:05.868121 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.868002 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede"} err="failed to get container status \"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede\": rpc error: code = NotFound desc = could not find container \"31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede\": container with ID starting with 31fc60f5b411be5a0258d6fe3e4db9d46554b4289ad7651560286ae38f395ede not found: ID does not exist" Jul 07 19:45:05.868121 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.868023 2568 scope.go:117] "RemoveContainer" containerID="94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064" Jul 07 19:45:05.868301 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:05.868279 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064\": container with ID starting with 94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064 not found: ID does not exist" containerID="94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064" Jul 07 19:45:05.868301 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.868302 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064"} err="failed to get container status \"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064\": rpc error: code = NotFound desc = could not find container \"94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064\": container with ID starting with 94bb121dbc3413f12e92b7a088cb365147e2f18906ecf93299c2cbd63359e064 not found: ID does not exist" Jul 07 19:45:05.871472 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.871443 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:45:05.878468 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.878395 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c47ca8-openshift-default-6459f9d55f-4lqf5"] Jul 07 19:45:05.902566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.902529 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed790476-9093-4deb-b493-5502ab395b5d-istiod-ca-cert\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:05.902566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:05.902555 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgtpr\" (UniqueName: \"kubernetes.io/projected/ed790476-9093-4deb-b493-5502ab395b5d-kube-api-access-kgtpr\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:07.352729 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:07.352696 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed790476-9093-4deb-b493-5502ab395b5d" path="/var/lib/kubelet/pods/ed790476-9093-4deb-b493-5502ab395b5d/volumes" Jul 07 19:45:08.328620 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:08.328573 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:08.786787 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:08.786753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:09.102620 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:09.102512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:09.227040 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:09.227013 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:09.317589 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:09.317549 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:45:12.028833 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.028797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:12.102511 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.102475 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:12.102992 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.102973 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.102994 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103020 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103028 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103046 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103056 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103077 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103085 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103164 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.103441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103175 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103182 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59780ee4-56f3-421d-ab6f-421a2d9350ad" containerName="istio-proxy" Jul 07 19:45:12.103441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.103188 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed790476-9093-4deb-b493-5502ab395b5d" containerName="istio-proxy" Jul 07 19:45:12.107755 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.107719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.110124 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.110095 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-b4b8d7-serving-cert\"" Jul 07 19:45:12.110292 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.110108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8pz6d\"" Jul 07 19:45:12.119673 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.119641 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:12.131740 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.131710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:12.163020 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.162974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.163219 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.163049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.163219 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.163096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65vm\" (UniqueName: \"kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.220461 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.220428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:12.264686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.264648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.264686 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.264704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65vm\" (UniqueName: \"kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.264955 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.264811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.264955 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:12.264813 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-b4b8d7-serving-cert: secret "maas-api-e2e-api-b-b4b8d7-serving-cert" not found Jul 07 19:45:12.265032 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:12.264987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls podName:a110c108-f5e1-41ef-8e5c-17bfccd16fcb nodeName:}" failed. No retries permitted until 2026-07-07 19:45:12.764962054 +0000 UTC m=+2158.115610269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls") pod "maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" (UID: "a110c108-f5e1-41ef-8e5c-17bfccd16fcb") : secret "maas-api-e2e-api-b-b4b8d7-serving-cert" not found Jul 07 19:45:12.265521 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.265499 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.274003 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.273973 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65vm\" (UniqueName: \"kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.702689 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.702651 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:12.770712 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.770668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.773353 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.773329 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-b4b8d7-568664db56-w24gg\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:12.800616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.800578 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:12.925033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:12.924971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:13.020647 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.020544 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:13.176732 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.176691 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:13.178290 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:45:13.178263 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda110c108_f5e1_41ef_8e5c_17bfccd16fcb.slice/crio-1c102f7f6bee5bedacef1d5c9841a75487b8e0e7e4ee8c09eeae6749e1b7f277 WatchSource:0}: Error finding container 1c102f7f6bee5bedacef1d5c9841a75487b8e0e7e4ee8c09eeae6749e1b7f277: Status 404 returned error can't find the container with id 1c102f7f6bee5bedacef1d5c9841a75487b8e0e7e4ee8c09eeae6749e1b7f277 Jul 07 19:45:13.180038 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.180020 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:45:13.903224 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.903126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" event={"ID":"a110c108-f5e1-41ef-8e5c-17bfccd16fcb","Type":"ContainerStarted","Data":"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b"} Jul 07 19:45:13.903224 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.903179 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" event={"ID":"a110c108-f5e1-41ef-8e5c-17bfccd16fcb","Type":"ContainerStarted","Data":"1c102f7f6bee5bedacef1d5c9841a75487b8e0e7e4ee8c09eeae6749e1b7f277"} Jul 07 19:45:13.903445 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.903232 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:13.929033 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:13.928978 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" podStartSLOduration=1.460947192 podStartE2EDuration="1.928955534s" totalImagesPullingTime="468.008342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:45:13.927052652 +0000 UTC m=+2159.277700922" watchObservedRunningTime="2026-07-07 19:45:13.928955534 +0000 UTC m=+2159.279603771" Jul 07 19:45:16.423725 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:16.423688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:19.218987 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:19.218942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:45:19.915001 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:19.914968 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:24.472855 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:24.472815 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:24.878695 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:24.878668 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:25.009499 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:25.009466 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:25.108228 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:25.108193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:25.296457 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:25.296347 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:30.814756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:30.814716 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:45:31.539252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:31.539221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:31.639312 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:31.639270 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:31.796254 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:31.796172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.040968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.040935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.206627 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.206594 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.595527 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.595439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.709395 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.709340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.808812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.808779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:32.961487 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.961451 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:32.961701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:32.961680 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" podUID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" containerName="maas-api" containerID="cri-o://2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b" gracePeriod=30 Jul 07 19:45:33.010096 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.010059 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:33.219965 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.219872 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:33.380859 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.380826 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") pod \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " Jul 07 19:45:33.381093 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.380963 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca\") pod \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " Jul 07 19:45:33.381093 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.381015 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q65vm\" (UniqueName: \"kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm\") pod \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\" (UID: \"a110c108-f5e1-41ef-8e5c-17bfccd16fcb\") " Jul 07 19:45:33.381341 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.381314 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a110c108-f5e1-41ef-8e5c-17bfccd16fcb" (UID: "a110c108-f5e1-41ef-8e5c-17bfccd16fcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:45:33.383086 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.383057 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm" (OuterVolumeSpecName: "kube-api-access-q65vm") pod "a110c108-f5e1-41ef-8e5c-17bfccd16fcb" (UID: "a110c108-f5e1-41ef-8e5c-17bfccd16fcb"). InnerVolumeSpecName "kube-api-access-q65vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:45:33.383205 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.383178 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a110c108-f5e1-41ef-8e5c-17bfccd16fcb" (UID: "a110c108-f5e1-41ef-8e5c-17bfccd16fcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:45:33.482416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.482318 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-openshift-service-ca\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:33.482416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.482351 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65vm\" (UniqueName: \"kubernetes.io/projected/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-kube-api-access-q65vm\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:33.482416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:33.482366 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a110c108-f5e1-41ef-8e5c-17bfccd16fcb-maas-api-tls\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:45:34.021498 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.021466 2568 generic.go:356] "Generic (PLEG): container finished" podID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" containerID="2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b" exitCode=0 Jul 07 19:45:34.021722 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.021513 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" Jul 07 19:45:34.021722 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.021529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" event={"ID":"a110c108-f5e1-41ef-8e5c-17bfccd16fcb","Type":"ContainerDied","Data":"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b"} Jul 07 19:45:34.021722 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.021581 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg" event={"ID":"a110c108-f5e1-41ef-8e5c-17bfccd16fcb","Type":"ContainerDied","Data":"1c102f7f6bee5bedacef1d5c9841a75487b8e0e7e4ee8c09eeae6749e1b7f277"} Jul 07 19:45:34.021722 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.021603 2568 scope.go:117] "RemoveContainer" containerID="2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b" Jul 07 19:45:34.031712 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.031693 2568 scope.go:117] "RemoveContainer" containerID="2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b" Jul 07 19:45:34.032031 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:34.032004 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b\": container with ID starting with 2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b not found: ID does not exist" containerID="2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b" Jul 07 19:45:34.032123 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.032043 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b"} err="failed to get container status \"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b\": rpc error: code = NotFound desc = could not find container \"2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b\": container with ID starting with 2b364af876bbc6735cbb7a59c0b94f1cc2fdc81c5a1b26dd63d7912490a6322b not found: ID does not exist" Jul 07 19:45:34.045709 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.045673 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:34.050549 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:34.050511 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b4b8d7-568664db56-w24gg"] Jul 07 19:45:35.354001 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:35.353962 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" path="/var/lib/kubelet/pods/a110c108-f5e1-41ef-8e5c-17bfccd16fcb/volumes" Jul 07 19:45:39.287137 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:39.287105 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:39.388761 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:39.388728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:39.567095 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:39.567014 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:39.784649 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:39.784619 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:40.124393 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:40.124363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:40.437200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:40.437105 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:40.679009 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:40.678970 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:42.211843 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:42.211800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:45:42.985171 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:42.985137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.052116 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.052084 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:45:43.052665 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.052651 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" containerName="maas-api" Jul 07 19:45:43.052718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.052668 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" containerName="maas-api" Jul 07 19:45:43.052823 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.052804 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a110c108-f5e1-41ef-8e5c-17bfccd16fcb" containerName="maas-api" Jul 07 19:45:43.057724 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.057699 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.060048 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.060022 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-8pz6d\"" Jul 07 19:45:43.060187 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.060033 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-78237e-serving-cert\"" Jul 07 19:45:43.066417 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.066377 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:45:43.068822 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.068793 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.072611 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.072576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.072756 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.072673 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj24\" (UniqueName: \"kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.072826 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.072778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.150796 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.150764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.174092 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.174058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj24\" (UniqueName: \"kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.174311 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.174142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.174311 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.174243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.174463 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:43.174376 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-78237e-serving-cert: secret "maas-api-e2e-shared-a-78237e-serving-cert" not found Jul 07 19:45:43.174463 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:45:43.174444 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls podName:d62ea75c-bcbd-401a-98d9-419738b33755 nodeName:}" failed. No retries permitted until 2026-07-07 19:45:43.674422747 +0000 UTC m=+2189.025070964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls") pod "maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" (UID: "d62ea75c-bcbd-401a-98d9-419738b33755") : secret "maas-api-e2e-shared-a-78237e-serving-cert" not found Jul 07 19:45:43.174841 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.174823 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.183964 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.183937 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj24\" (UniqueName: \"kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.299184 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.299092 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.678405 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.678363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.680799 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.680780 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") pod \"maas-api-e2e-shared-a-78237e-8797f84d5-cvztd\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:43.718995 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.718959 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.822678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.822639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:43.971362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:43.971255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:44.119032 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:44.119004 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:45:44.120928 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:45:44.120896 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62ea75c_bcbd_401a_98d9_419738b33755.slice/crio-7102b13fde325b22f04ee8bb8e6eb38106cf88c2ffa0dd416cdc662200c0ba7e WatchSource:0}: Error finding container 7102b13fde325b22f04ee8bb8e6eb38106cf88c2ffa0dd416cdc662200c0ba7e: Status 404 returned error can't find the container with id 7102b13fde325b22f04ee8bb8e6eb38106cf88c2ffa0dd416cdc662200c0ba7e Jul 07 19:45:45.087094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:45.087063 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" event={"ID":"d62ea75c-bcbd-401a-98d9-419738b33755","Type":"ContainerStarted","Data":"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e"} Jul 07 19:45:45.087496 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:45.087111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" event={"ID":"d62ea75c-bcbd-401a-98d9-419738b33755","Type":"ContainerStarted","Data":"7102b13fde325b22f04ee8bb8e6eb38106cf88c2ffa0dd416cdc662200c0ba7e"} Jul 07 19:45:45.087496 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:45.087142 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:45.105615 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:45.105565 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" podStartSLOduration=1.7420464089999999 podStartE2EDuration="2.105547607s" totalImagesPullingTime="363.501198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:45:45.104760761 +0000 UTC m=+2190.455409009" watchObservedRunningTime="2026-07-07 19:45:45.105547607 +0000 UTC m=+2190.456195844" Jul 07 19:45:47.504462 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:47.504429 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:47.929264 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:47.929231 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:48.173530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:48.173490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:48.789022 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:48.788988 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.054135 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.054097 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.099147 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.099112 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:45:51.163227 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.163194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.339432 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.339338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.472414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.472386 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.815678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.815639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:51.910905 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.910867 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:45:51.914433 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:51.914394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:45:55.558836 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:45:55.558807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:00.404636 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:00.404593 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:46:01.717441 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:01.717401 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:04.609962 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:04.609930 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:04.727021 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:04.726982 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:07.515725 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:07.515684 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:46:12.612880 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:12.612829 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:12.712416 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:12.712380 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:43.249830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:43.249750 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:46:46.712471 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:46:46.712426 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:47:04.208602 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:47:04.208560 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:47:19.809608 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:47:19.809566 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:47:35.508718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:47:35.508676 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:47:47.526335 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:47:47.526298 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:47:52.022844 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:47:52.022806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:48:09.508306 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:09.508218 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:48:24.948588 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:24.948550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:26.159460 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:26.159425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:26.610732 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:26.610695 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:48:40.089271 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:40.089234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:41.068270 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:41.068236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:43.117952 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:43.117895 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:48:44.149541 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:44.149498 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:44.250199 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:44.250166 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:48:56.409074 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:48:56.409031 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:49:00.761585 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:49:00.761546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:04.511845 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:49:04.511800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:49:23.557897 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:49:23.557858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:49:27.806678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:49:27.806637 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:49:56.811801 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:49:56.811757 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:50:01.998084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:01.998051 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:13.903102 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:13.903061 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:50:20.758718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:20.758684 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:21.718733 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:21.718699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:29.940214 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:29.940176 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:50:32.818786 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:50:32.818750 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:51:01.337584 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:01.337546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:05.218162 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:05.218122 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:51:05.483104 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:05.483019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:05.581364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:05.581311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:06.527997 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:06.527876 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:06.628014 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:06.627979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:13.368623 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:13.368583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:13.468119 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:13.468082 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:31.508822 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:31.508783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:51:44.506566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:44.506526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:51:50.636081 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:50.636045 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:51.515881 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:51.515847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:57.059519 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.059481 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:57.316863 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.316778 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:51:57.458568 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.458536 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:57.630365 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.630332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:57.730688 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.730632 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:57.865467 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.865421 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:51:57.865678 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.865652 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" podUID="d62ea75c-bcbd-401a-98d9-419738b33755" containerName="maas-api" containerID="cri-o://42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e" gracePeriod=30 Jul 07 19:51:57.893462 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:57.893413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:51:58.131050 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.131021 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:51:58.247830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.247733 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") pod \"d62ea75c-bcbd-401a-98d9-419738b33755\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " Jul 07 19:51:58.247830 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.247783 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca\") pod \"d62ea75c-bcbd-401a-98d9-419738b33755\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " Jul 07 19:51:58.248078 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.247861 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxj24\" (UniqueName: \"kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24\") pod \"d62ea75c-bcbd-401a-98d9-419738b33755\" (UID: \"d62ea75c-bcbd-401a-98d9-419738b33755\") " Jul 07 19:51:58.248236 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.248210 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d62ea75c-bcbd-401a-98d9-419738b33755" (UID: "d62ea75c-bcbd-401a-98d9-419738b33755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:51:58.250084 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.250054 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d62ea75c-bcbd-401a-98d9-419738b33755" (UID: "d62ea75c-bcbd-401a-98d9-419738b33755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:51:58.250228 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.250206 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24" (OuterVolumeSpecName: "kube-api-access-lxj24") pod "d62ea75c-bcbd-401a-98d9-419738b33755" (UID: "d62ea75c-bcbd-401a-98d9-419738b33755"). InnerVolumeSpecName "kube-api-access-lxj24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:51:58.280616 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.280584 2568 generic.go:356] "Generic (PLEG): container finished" podID="d62ea75c-bcbd-401a-98d9-419738b33755" containerID="42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e" exitCode=0 Jul 07 19:51:58.280775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.280634 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" Jul 07 19:51:58.280775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.280659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" event={"ID":"d62ea75c-bcbd-401a-98d9-419738b33755","Type":"ContainerDied","Data":"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e"} Jul 07 19:51:58.280775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.280712 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd" event={"ID":"d62ea75c-bcbd-401a-98d9-419738b33755","Type":"ContainerDied","Data":"7102b13fde325b22f04ee8bb8e6eb38106cf88c2ffa0dd416cdc662200c0ba7e"} Jul 07 19:51:58.280775 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.280734 2568 scope.go:117] "RemoveContainer" containerID="42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e" Jul 07 19:51:58.290452 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.290426 2568 scope.go:117] "RemoveContainer" containerID="42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e" Jul 07 19:51:58.290738 ip-10-0-142-116 kubenswrapper[2568]: E0707 19:51:58.290717 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e\": container with ID starting with 42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e not found: ID does not exist" containerID="42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e" Jul 07 19:51:58.290812 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.290746 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e"} err="failed to get container status \"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e\": rpc error: code = NotFound desc = could not find container \"42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e\": container with ID starting with 42e08aef9832bf51a182a3da874b4fdea11289c45eb81ce6d7fbb2662b696e9e not found: ID does not exist" Jul 07 19:51:58.303659 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.303624 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:51:58.308333 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.308298 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-78237e-8797f84d5-cvztd"] Jul 07 19:51:58.349154 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.349113 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62ea75c-bcbd-401a-98d9-419738b33755-maas-api-tls\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:51:58.349154 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.349145 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62ea75c-bcbd-401a-98d9-419738b33755-openshift-service-ca\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:51:58.349154 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:58.349156 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj24\" (UniqueName: \"kubernetes.io/projected/d62ea75c-bcbd-401a-98d9-419738b33755-kube-api-access-lxj24\") on node \"ip-10-0-142-116.ec2.internal\" DevicePath \"\"" Jul 07 19:51:59.359238 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:51:59.359198 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62ea75c-bcbd-401a-98d9-419738b33755" path="/var/lib/kubelet/pods/d62ea75c-bcbd-401a-98d9-419738b33755/volumes" Jul 07 19:52:02.535266 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:02.535236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:06.898094 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:06.898054 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:52:08.858625 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:08.858592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:09.137939 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:09.137835 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:09.355161 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:09.355126 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:09.556578 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:09.556480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:09.937753 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:09.937720 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:10.039008 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:10.038974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:10.141105 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:10.141071 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:10.328657 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:10.328566 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:16.712631 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:16.712591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-j9c2k"] Jul 07 19:52:18.152843 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:18.152813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:18.274587 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:18.274553 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:18.470390 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:18.470303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:18.665864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:18.665830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:19.098080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:19.098045 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658c6b66d5-2bvg9_e9440065-e959-4c0b-a0cd-8015de89e0e8/authorino/0.log" Jul 07 19:52:23.901758 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:23.901728 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gz2s2_b4be1f47-6af1-4444-8f7f-de7210d97298/manager/0.log" Jul 07 19:52:24.034413 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:24.034379 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kmv9p_dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7/manager/0.log" Jul 07 19:52:24.153002 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:24.152898 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69775b867f-jptc7_ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e/maas-api/0.log" Jul 07 19:52:24.267316 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:24.267288 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724210-xwjv2_7aab2161-05dc-4edd-bc8c-bd89b6941cf4/cleanup/0.log" Jul 07 19:52:24.750371 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:24.750338 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t6qj6_9b946389-9f25-40f3-ba29-b4da40029811/manager/0.log" Jul 07 19:52:25.002566 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:25.002480 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75847b84df-hccdw_8e1eba71-1c5d-457e-8878-7b6a3fb5ff56/copy-manifests/0.log" Jul 07 19:52:25.009677 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:25.009651 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75847b84df-hccdw_8e1eba71-1c5d-457e-8878-7b6a3fb5ff56/manager/0.log" Jul 07 19:52:26.020031 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.020002 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/util/0.log" Jul 07 19:52:26.031109 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.031081 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/pull/0.log" Jul 07 19:52:26.039036 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.039010 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/extract/0.log" Jul 07 19:52:26.146815 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.146785 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/util/0.log" Jul 07 19:52:26.154185 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.154157 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/pull/0.log" Jul 07 19:52:26.161481 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.161451 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/extract/0.log" Jul 07 19:52:26.266440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.266417 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/extract/0.log" Jul 07 19:52:26.274530 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.274450 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/util/0.log" Jul 07 19:52:26.281899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.281879 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/pull/0.log" Jul 07 19:52:26.390389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.390358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/util/0.log" Jul 07 19:52:26.398652 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.398625 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/pull/0.log" Jul 07 19:52:26.407577 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.407556 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/extract/0.log" Jul 07 19:52:26.519723 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.519690 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658c6b66d5-2bvg9_e9440065-e959-4c0b-a0cd-8015de89e0e8/authorino/0.log" Jul 07 19:52:26.638968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.638907 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-57vn5_a88e8519-3d49-40bc-bfb3-bd6d63f02df0/manager/0.log" Jul 07 19:52:26.744878 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.744849 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sjr2t_c4e19632-bc87-47d6-9c71-793768ad895d/manager/0.log" Jul 07 19:52:26.969269 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:26.969188 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lg245_0a419f62-2cc2-42a1-8e64-f00bbee05709/registry-server/0.log" Jul 07 19:52:27.213066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:27.213039 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-j9c2k_631ca361-3284-4640-abcf-47cf8ee18053/limitador/0.log" Jul 07 19:52:27.348058 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:27.347979 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-v48sp_66984758-bbec-48b1-8422-eb0fcd699464/manager/0.log" Jul 07 19:52:27.683968 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:27.683939 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qdrdw_ad5ae484-227d-4297-a00c-303a4cce288b/istio-proxy/0.log" Jul 07 19:52:27.916200 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:27.916174 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-598d7cb498-4c7nh_58177ee8-930d-43ed-8050-3ad7fbd97d26/kube-auth-proxy/0.log" Jul 07 19:52:28.147911 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:28.147878 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/1.log" Jul 07 19:52:28.827701 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:28.827670 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wzvcw_f78193ed-8be6-4792-b06e-6895dda565e7/main/0.log" Jul 07 19:52:33.181895 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.181857 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j2nl9/must-gather-hh565"] Jul 07 19:52:33.182332 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.182319 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62ea75c-bcbd-401a-98d9-419738b33755" containerName="maas-api" Jul 07 19:52:33.182380 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.182333 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62ea75c-bcbd-401a-98d9-419738b33755" containerName="maas-api" Jul 07 19:52:33.182422 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.182413 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62ea75c-bcbd-401a-98d9-419738b33755" containerName="maas-api" Jul 07 19:52:33.185049 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.185031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.187427 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.187399 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2nl9\"/\"openshift-service-ca.crt\"" Jul 07 19:52:33.187590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.187399 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j2nl9\"/\"default-dockercfg-qqkct\"" Jul 07 19:52:33.187590 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.187403 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j2nl9\"/\"kube-root-ca.crt\"" Jul 07 19:52:33.196436 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.196406 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2nl9/must-gather-hh565"] Jul 07 19:52:33.216134 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.216103 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:33.286211 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.286159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27v8f\" (UniqueName: \"kubernetes.io/projected/32975779-9cf1-40e7-818f-a96c832ab2f4-kube-api-access-27v8f\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.286414 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.286342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32975779-9cf1-40e7-818f-a96c832ab2f4-must-gather-output\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.319322 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.319288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" limit=1024 Jul 07 19:52:33.387355 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.387318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32975779-9cf1-40e7-818f-a96c832ab2f4-must-gather-output\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.387355 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.387364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27v8f\" (UniqueName: \"kubernetes.io/projected/32975779-9cf1-40e7-818f-a96c832ab2f4-kube-api-access-27v8f\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.387675 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.387657 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32975779-9cf1-40e7-818f-a96c832ab2f4-must-gather-output\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.397175 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.397147 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27v8f\" (UniqueName: \"kubernetes.io/projected/32975779-9cf1-40e7-818f-a96c832ab2f4-kube-api-access-27v8f\") pod \"must-gather-hh565\" (UID: \"32975779-9cf1-40e7-818f-a96c832ab2f4\") " pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.495864 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.495776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2nl9/must-gather-hh565" Jul 07 19:52:33.653645 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.653563 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2nl9/must-gather-hh565"] Jul 07 19:52:33.656188 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:52:33.656154 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32975779_9cf1_40e7_818f_a96c832ab2f4.slice/crio-be7cbfb2904c1cc0de0a820d0505800898959aac569c857f8d694bf4f2a79d2d WatchSource:0}: Error finding container be7cbfb2904c1cc0de0a820d0505800898959aac569c857f8d694bf4f2a79d2d: Status 404 returned error can't find the container with id be7cbfb2904c1cc0de0a820d0505800898959aac569c857f8d694bf4f2a79d2d Jul 07 19:52:33.658103 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:33.658083 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:52:34.485066 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:34.485030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2nl9/must-gather-hh565" event={"ID":"32975779-9cf1-40e7-818f-a96c832ab2f4","Type":"ContainerStarted","Data":"be7cbfb2904c1cc0de0a820d0505800898959aac569c857f8d694bf4f2a79d2d"} Jul 07 19:52:35.494899 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:35.494862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2nl9/must-gather-hh565" event={"ID":"32975779-9cf1-40e7-818f-a96c832ab2f4","Type":"ContainerStarted","Data":"c8d1a664b4b3e8564ce188195dc13cb40433f3033158438e5c101c27a0778957"} Jul 07 19:52:35.495364 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:35.494908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2nl9/must-gather-hh565" event={"ID":"32975779-9cf1-40e7-818f-a96c832ab2f4","Type":"ContainerStarted","Data":"9766ce135ed1b0dc71db707377526056500a60aa94e024f49811f2a3c0bd33ac"} Jul 07 19:52:35.516073 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:35.516006 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j2nl9/must-gather-hh565" podStartSLOduration=1.4122006520000001 podStartE2EDuration="2.515981331s" totalImagesPullingTime="1.103780679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:52:35.511887944 +0000 UTC m=+2600.862536181" watchObservedRunningTime="2026-07-07 19:52:35.515981331 +0000 UTC m=+2600.866629569" Jul 07 19:52:36.533989 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:36.533899 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g4dct_134d60c9-28cc-4320-ac6b-740976d6382d/global-pull-secret-syncer/0.log" Jul 07 19:52:36.689394 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:36.689353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h7g62_c1535d4c-d58e-4fcc-addd-65cd866a41bd/konnectivity-agent/0.log" Jul 07 19:52:36.847712 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:36.847603 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-116.ec2.internal_85dca3ea545cba1033034b0f993d4dd5/haproxy/0.log" Jul 07 19:52:41.544362 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.544325 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/extract/0.log" Jul 07 19:52:41.577778 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.577745 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/util/0.log" Jul 07 19:52:41.611425 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.611386 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759842d6_00cba973-55fe-46c2-982d-96b2c9630179/pull/0.log" Jul 07 19:52:41.658938 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.658546 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/extract/0.log" Jul 07 19:52:41.699635 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.699592 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/util/0.log" Jul 07 19:52:41.735417 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.735368 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7sz4_da3daa69-2f5b-497e-8e4d-f1c2ec6db037/pull/0.log" Jul 07 19:52:41.778747 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.778717 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/extract/0.log" Jul 07 19:52:41.824971 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.824844 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/util/0.log" Jul 07 19:52:41.862777 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.862741 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sntgb_2cc51cc1-7e44-4aff-a809-0cf8daab9e35/pull/0.log" Jul 07 19:52:41.899082 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.899018 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/extract/0.log" Jul 07 19:52:41.929998 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.929973 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/util/0.log" Jul 07 19:52:41.963526 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:41.963499 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kgnmk_545295a5-64a6-49ca-8134-565684dcb9e6/pull/0.log" Jul 07 19:52:42.335440 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.335402 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658c6b66d5-2bvg9_e9440065-e959-4c0b-a0cd-8015de89e0e8/authorino/0.log" Jul 07 19:52:42.379026 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.378988 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-57vn5_a88e8519-3d49-40bc-bfb3-bd6d63f02df0/manager/0.log" Jul 07 19:52:42.413891 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.413840 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-sjr2t_c4e19632-bc87-47d6-9c71-793768ad895d/manager/0.log" Jul 07 19:52:42.502750 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.502713 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lg245_0a419f62-2cc2-42a1-8e64-f00bbee05709/registry-server/0.log" Jul 07 19:52:42.733687 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.733659 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-j9c2k_631ca361-3284-4640-abcf-47cf8ee18053/limitador/0.log" Jul 07 19:52:42.833424 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:42.833392 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-v48sp_66984758-bbec-48b1-8422-eb0fcd699464/manager/0.log" Jul 07 19:52:44.553575 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.553532 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/alertmanager/0.log" Jul 07 19:52:44.585981 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.585950 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/config-reloader/0.log" Jul 07 19:52:44.620265 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.620229 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/kube-rbac-proxy-web/0.log" Jul 07 19:52:44.654604 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.654552 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/kube-rbac-proxy/0.log" Jul 07 19:52:44.705152 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.705117 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/kube-rbac-proxy-metric/0.log" Jul 07 19:52:44.735891 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.735862 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/prom-label-proxy/0.log" Jul 07 19:52:44.778248 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.778185 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8cdbed3b-5378-4764-af0e-a061df9c7db8/init-config-reloader/0.log" Jul 07 19:52:44.939258 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.939232 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78d97859f4-lfmk5_668972cb-27df-40f5-ab4b-26399cc3c110/metrics-server/0.log" Jul 07 19:52:44.972387 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:44.972352 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-9lnx5_1b2a40ba-6e3c-4569-a372-4c2d74655d15/monitoring-plugin/0.log" Jul 07 19:52:45.211857 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.211740 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptdb8_416a3b96-aba2-45b6-a8fd-5ec7cd21db11/node-exporter/0.log" Jul 07 19:52:45.240341 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.240307 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptdb8_416a3b96-aba2-45b6-a8fd-5ec7cd21db11/kube-rbac-proxy/0.log" Jul 07 19:52:45.270599 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.270573 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptdb8_416a3b96-aba2-45b6-a8fd-5ec7cd21db11/init-textfile/0.log" Jul 07 19:52:45.614258 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.614217 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-4m5qs_728513c9-ccf1-4591-8a05-43988359a583/prometheus-operator/0.log" Jul 07 19:52:45.641223 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.641189 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-4m5qs_728513c9-ccf1-4591-8a05-43988359a583/kube-rbac-proxy/0.log" Jul 07 19:52:45.715230 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.715187 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bb8587-gpsnc_df8468d8-2ba6-4cb0-b571-919ae29116f7/telemeter-client/0.log" Jul 07 19:52:45.744232 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.744202 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bb8587-gpsnc_df8468d8-2ba6-4cb0-b571-919ae29116f7/reload/0.log" Jul 07 19:52:45.787172 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:45.787148 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74bb8587-gpsnc_df8468d8-2ba6-4cb0-b571-919ae29116f7/kube-rbac-proxy/0.log" Jul 07 19:52:46.720900 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.720861 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x"] Jul 07 19:52:46.726910 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.726883 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.735658 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.735038 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x"] Jul 07 19:52:46.835322 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.835287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-podres\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.835506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.835404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-proc\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.835506 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.835440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-sys\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.835599 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.835567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-lib-modules\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.835650 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.835631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjv8\" (UniqueName: \"kubernetes.io/projected/b6bb0d87-0417-4b64-b79e-af0abfed3d71-kube-api-access-7jjv8\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937087 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-lib-modules\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjv8\" (UniqueName: \"kubernetes.io/projected/b6bb0d87-0417-4b64-b79e-af0abfed3d71-kube-api-access-7jjv8\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-podres\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937206 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-lib-modules\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937283 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-proc\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-proc\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-podres\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-sys\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.937495 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.937339 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6bb0d87-0417-4b64-b79e-af0abfed3d71-sys\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:46.946948 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:46.946879 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjv8\" (UniqueName: \"kubernetes.io/projected/b6bb0d87-0417-4b64-b79e-af0abfed3d71-kube-api-access-7jjv8\") pod \"perf-node-gather-daemonset-lh96x\" (UID: \"b6bb0d87-0417-4b64-b79e-af0abfed3d71\") " pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:47.040518 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.040413 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:47.120247 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.120186 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-w4tfx_c08a6d92-0b57-4e67-9c9a-271e6b51755c/networking-console-plugin/0.log" Jul 07 19:52:47.232090 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.231320 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x"] Jul 07 19:52:47.234519 ip-10-0-142-116 kubenswrapper[2568]: W0707 19:52:47.234456 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb6bb0d87_0417_4b64_b79e_af0abfed3d71.slice/crio-a97447e6663411ad41d22579bf1ce2c08664cb4e190221fe1f7ae6cc734b3bd5 WatchSource:0}: Error finding container a97447e6663411ad41d22579bf1ce2c08664cb4e190221fe1f7ae6cc734b3bd5: Status 404 returned error can't find the container with id a97447e6663411ad41d22579bf1ce2c08664cb4e190221fe1f7ae6cc734b3bd5 Jul 07 19:52:47.593252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.593161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" event={"ID":"b6bb0d87-0417-4b64-b79e-af0abfed3d71","Type":"ContainerStarted","Data":"3081a82db859523c83c68e25373b0fc7e830a50fb2704794a02e5f3132a3757b"} Jul 07 19:52:47.593252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.593221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" event={"ID":"b6bb0d87-0417-4b64-b79e-af0abfed3d71","Type":"ContainerStarted","Data":"a97447e6663411ad41d22579bf1ce2c08664cb4e190221fe1f7ae6cc734b3bd5"} Jul 07 19:52:47.593252 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.593251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:47.613238 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:47.613178 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" podStartSLOduration=1.6131576189999999 podStartE2EDuration="1.613157619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:52:47.612514602 +0000 UTC m=+2612.963162841" watchObservedRunningTime="2026-07-07 19:52:47.613157619 +0000 UTC m=+2612.963805855" Jul 07 19:52:48.904497 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:48.904464 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-nfdcm_b75f30fe-99ef-46e4-90c6-6ba38e4ec27c/volume-data-source-validator/0.log" Jul 07 19:52:50.126176 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:50.126141 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twdjg_4b80486a-4598-4c45-a666-6be9da1b63d9/dns/0.log" Jul 07 19:52:50.155102 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:50.155070 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twdjg_4b80486a-4598-4c45-a666-6be9da1b63d9/kube-rbac-proxy/0.log" Jul 07 19:52:50.270491 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:50.270468 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2p956_6e39206e-6b9b-4a92-a0ed-705902137faf/dns-node-resolver/0.log" Jul 07 19:52:50.883711 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:50.883649 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f85858d46-w926r_25a5a89f-b332-4fc6-a920-f6ae0ec6aa31/registry/0.log" Jul 07 19:52:50.973560 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:50.973525 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xhfcg_2dfb8567-19e4-48fa-98a5-68e65b8a3378/node-ca/0.log" Jul 07 19:52:52.089937 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:52.089900 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qdrdw_ad5ae484-227d-4297-a00c-303a4cce288b/istio-proxy/0.log" Jul 07 19:52:52.245053 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:52.245015 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-598d7cb498-4c7nh_58177ee8-930d-43ed-8050-3ad7fbd97d26/kube-auth-proxy/0.log" Jul 07 19:52:52.311507 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:52.311479 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/0.log" Jul 07 19:52:52.348080 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:52.347998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/1.log" Jul 07 19:52:53.279063 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:53.279012 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wln8w_48d69b72-f6ff-4c8b-b14d-fe19db1eb685/serve-healthcheck-canary/0.log" Jul 07 19:52:53.617007 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:53.615993 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j2nl9/perf-node-gather-daemonset-lh96x" Jul 07 19:52:53.849883 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:53.849851 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-b2pvv_a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758/insights-operator/0.log" Jul 07 19:52:53.850368 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:53.850346 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-b2pvv_a1a8edb3-ae0b-4a1d-8e3b-dcd05588d758/insights-operator/1.log" Jul 07 19:52:54.087025 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:54.086886 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njvmk_da03f687-4e8f-4302-bbc4-b203e1862856/kube-rbac-proxy/0.log" Jul 07 19:52:54.125136 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:54.125098 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njvmk_da03f687-4e8f-4302-bbc4-b203e1862856/exporter/0.log" Jul 07 19:52:54.157053 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:54.157023 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njvmk_da03f687-4e8f-4302-bbc4-b203e1862856/extractor/0.log" Jul 07 19:52:56.623834 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:56.623805 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gz2s2_b4be1f47-6af1-4444-8f7f-de7210d97298/manager/0.log" Jul 07 19:52:56.821213 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:56.821067 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kmv9p_dbcdf5de-94d7-4c6f-bb0a-2a5510e7ade7/manager/0.log" Jul 07 19:52:56.877676 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:56.877624 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69775b867f-jptc7_ecfe8aa0-6656-493b-82f0-a64d3fe0ef5e/maas-api/0.log" Jul 07 19:52:56.906269 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:56.906235 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724210-xwjv2_7aab2161-05dc-4edd-bc8c-bd89b6941cf4/cleanup/0.log" Jul 07 19:52:57.191947 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.190669 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t6qj6_9b946389-9f25-40f3-ba29-b4da40029811/manager/0.log" Jul 07 19:52:57.317363 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.317319 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qdrdw_ad5ae484-227d-4297-a00c-303a4cce288b/istio-proxy/0.log" Jul 07 19:52:57.342896 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.342869 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75847b84df-hccdw_8e1eba71-1c5d-457e-8878-7b6a3fb5ff56/manager/0.log" Jul 07 19:52:57.374222 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.374195 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75847b84df-hccdw_8e1eba71-1c5d-457e-8878-7b6a3fb5ff56/copy-manifests/0.log" Jul 07 19:52:57.445718 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.445632 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-598d7cb498-4c7nh_58177ee8-930d-43ed-8050-3ad7fbd97d26/kube-auth-proxy/0.log" Jul 07 19:52:57.497357 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.497318 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/0.log" Jul 07 19:52:57.534819 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:57.534776 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/1.log" Jul 07 19:52:58.988669 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:58.988638 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qdrdw_ad5ae484-227d-4297-a00c-303a4cce288b/istio-proxy/0.log" Jul 07 19:52:59.055056 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:59.054955 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-t4f8h_6b129705-0072-40fe-9af9-6bb49eef1167/openshift-lws-operator/0.log" Jul 07 19:52:59.133389 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:59.133359 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-598d7cb498-4c7nh_58177ee8-930d-43ed-8050-3ad7fbd97d26/kube-auth-proxy/0.log" Jul 07 19:52:59.190037 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:59.190004 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/0.log" Jul 07 19:52:59.226241 ip-10-0-142-116 kubenswrapper[2568]: I0707 19:52:59.226205 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qzk2g_b00a4dcf-b3fd-4a2b-bf94-bb96feda2147/istio-proxy/1.log"