Jul 09 16:19:05.346288 ip-10-0-133-162 systemd[1]: Starting Kubernetes Kubelet... Jul 09 16:19:05.786572 ip-10-0-133-162 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:19:05.786572 ip-10-0-133-162 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 16:19:05.786572 ip-10-0-133-162 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:19:05.786572 ip-10-0-133-162 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 16:19:05.786572 ip-10-0-133-162 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:19:05.788614 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.788520 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796102 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796136 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796140 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796144 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796146 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.796138 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796149 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796152 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796157 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796159 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796162 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796165 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796168 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796170 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796173 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796176 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796180 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796182 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796185 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796187 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796190 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796192 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796195 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796198 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796200 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796203 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.796372 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796205 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796208 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796217 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796220 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796223 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796226 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796228 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796231 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796233 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796236 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796239 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796242 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796244 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796247 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796250 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796253 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796255 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796258 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796260 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796263 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.796854 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796266 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796268 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796270 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796273 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796276 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796278 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796280 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796283 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796286 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796288 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796291 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796293 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796296 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796298 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796300 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796303 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796305 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796308 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796312 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796315 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.797354 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796318 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796321 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796324 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796326 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796330 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796333 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796336 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796338 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796341 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796345 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796348 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796352 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796355 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796358 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796361 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796364 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796367 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796370 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796379 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796383 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.797820 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796386 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796389 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796392 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796394 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796397 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796400 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796409 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796412 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796414 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796417 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796420 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796422 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796426 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.796430 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797205 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797214 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797218 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797221 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797225 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.798312 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797227 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797230 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797233 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797235 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797238 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797241 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797245 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797249 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797252 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797255 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797257 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797260 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797263 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797266 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797268 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797271 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797273 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797276 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797278 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797281 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.798769 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797284 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797286 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797289 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797291 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797294 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797296 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797299 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797301 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797303 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797306 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797309 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797311 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797314 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797316 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797320 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797323 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797327 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797330 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797332 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.799280 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797335 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797338 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797342 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797345 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797350 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797354 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797358 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797361 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797364 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797367 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797370 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797372 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797375 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797377 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797382 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797384 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797387 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797389 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797392 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.799745 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797395 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797397 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797400 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797402 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797405 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797407 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797410 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797412 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797415 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797417 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797420 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797422 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797425 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797429 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797431 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797433 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797436 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797440 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797443 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797445 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.800273 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797448 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797451 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797453 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797456 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797459 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797462 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797465 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797469 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797471 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797474 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797477 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797479 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797482 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797484 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797487 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797489 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797887 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797892 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797895 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.800753 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797899 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797903 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797906 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797909 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797912 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797932 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797936 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797941 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797945 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797949 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797953 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797955 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797958 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797961 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797964 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797966 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797969 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797971 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797974 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797977 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.801242 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797981 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797984 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797987 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797989 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797992 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797994 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797996 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.797999 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798001 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798004 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798006 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798009 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798011 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798014 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798016 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798018 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798021 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798023 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798026 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798028 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.801764 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798030 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798032 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798036 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798038 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798041 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798043 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798046 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798049 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798051 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798054 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798056 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798059 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798063 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798066 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798069 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798072 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798079 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798081 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798084 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.802259 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798088 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798091 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798093 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798095 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798098 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798100 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798103 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798105 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798107 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798110 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798112 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798115 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798117 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798120 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798122 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798125 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798127 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798130 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798132 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798135 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.802715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798138 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798141 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798143 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798145 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798148 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798153 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798157 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798160 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798162 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798165 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798168 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798170 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798173 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798176 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798178 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798181 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798183 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798262 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798278 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798285 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 16:19:05.803253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798289 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798293 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798296 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798300 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798305 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798308 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798311 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798315 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798318 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798321 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798324 2577 flags.go:64] FLAG: --cgroup-root="" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798327 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798330 2577 flags.go:64] FLAG: --client-ca-file="" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798334 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798337 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798341 2577 flags.go:64] FLAG: --cluster-domain="" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798344 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798349 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798352 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798355 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798359 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798362 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798365 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 16:19:05.803736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798368 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798371 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798374 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798377 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798380 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798382 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798386 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798389 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798392 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798395 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798398 2577 flags.go:64] FLAG: --enable-server="true" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798401 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798408 2577 flags.go:64] FLAG: --event-burst="100" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798411 2577 flags.go:64] FLAG: --event-qps="50" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798414 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798417 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798419 2577 flags.go:64] FLAG: --eviction-hard="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798423 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798426 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798429 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798432 2577 flags.go:64] FLAG: --eviction-soft="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798435 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798437 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798441 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798444 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 16:19:05.804310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798447 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798451 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798453 2577 flags.go:64] FLAG: --feature-gates="" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798457 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798460 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798463 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798466 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798469 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798472 2577 flags.go:64] FLAG: --help="false" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798475 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-162.ec2.internal" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798478 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798481 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798484 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798487 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798490 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798493 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798496 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798499 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798501 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798505 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798508 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798511 2577 flags.go:64] FLAG: --kube-reserved="" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798514 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798516 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 16:19:05.804909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798519 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798522 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798526 2577 flags.go:64] FLAG: --lock-file="" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798529 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798531 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798534 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798539 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798543 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798546 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798550 2577 flags.go:64] FLAG: --logging-format="text" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798553 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798556 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798558 2577 flags.go:64] FLAG: --manifest-url="" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798561 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798565 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798568 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798572 2577 flags.go:64] FLAG: --max-pods="110" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798575 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798579 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798581 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798584 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798587 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798590 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798593 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798600 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 16:19:05.805511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798602 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798605 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798609 2577 flags.go:64] FLAG: --pod-cidr="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798612 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798618 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798621 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798624 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798627 2577 flags.go:64] FLAG: --port="10250" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798630 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798632 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-043b771d4c7f3b1cc" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798636 2577 flags.go:64] FLAG: --qos-reserved="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798638 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798642 2577 flags.go:64] FLAG: --register-node="true" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798644 2577 flags.go:64] FLAG: --register-with-taints="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798648 2577 flags.go:64] FLAG: --registry-burst="10" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798653 2577 flags.go:64] FLAG: --registry-qps="5" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798657 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798660 2577 flags.go:64] FLAG: --reserved-memory="" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798663 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798666 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798669 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798672 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798674 2577 flags.go:64] FLAG: --runonce="false" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798677 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798680 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 16:19:05.806168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798683 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798686 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798689 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798692 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798695 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798698 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798701 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798703 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798706 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798709 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798714 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798717 2577 flags.go:64] FLAG: --system-cgroups="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798719 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798724 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798727 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798730 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798734 2577 flags.go:64] FLAG: --tls-min-version="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798737 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798740 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798743 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798745 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798748 2577 flags.go:64] FLAG: --v="2" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798752 2577 flags.go:64] FLAG: --version="false" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798759 2577 flags.go:64] FLAG: --vmodule="" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798763 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 16:19:05.806793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.798766 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798885 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798889 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798891 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798894 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798897 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798899 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798901 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798904 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798907 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798909 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798912 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798934 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798937 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798939 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798942 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798945 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798949 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798951 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798954 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.807405 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798956 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798959 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798961 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798964 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798967 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798969 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798972 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798974 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798977 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798979 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798983 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798987 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798989 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798992 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798994 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.798997 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799000 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799002 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799004 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799007 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.807898 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799009 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799012 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799014 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799017 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799019 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799021 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799024 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799026 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799029 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799033 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799036 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799038 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799040 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799043 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799045 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799048 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799050 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799053 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799055 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799057 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.808398 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799060 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799062 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799069 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799073 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799077 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799081 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799084 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799087 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799090 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799094 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799097 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799100 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799103 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799106 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799109 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799120 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799123 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799126 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799129 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.808878 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799132 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799134 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799139 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799142 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799145 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799148 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799151 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799154 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799157 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799160 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799163 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799166 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799170 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799173 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799176 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799180 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799183 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799185 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799189 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.809462 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799191 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.799194 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.800034 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.807413 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.807429 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.807447 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.807453 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807511 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807515 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807519 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807522 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807525 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807528 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807531 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.809897 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807533 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807536 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807538 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807541 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807544 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807546 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807549 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807551 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807553 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807556 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807558 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807561 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807564 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807566 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807568 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807571 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807575 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807580 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807583 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807586 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.810268 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807589 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807592 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807595 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807598 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807600 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807604 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807607 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807610 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807613 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807616 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807618 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807620 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807623 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807626 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807628 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807631 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807633 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807636 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807638 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.810815 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807640 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807643 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807645 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807648 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807650 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807652 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807655 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807658 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807660 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807663 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807665 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807668 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807671 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807674 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807676 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807679 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807682 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807685 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807687 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807689 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.811308 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807692 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807694 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807697 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807699 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807701 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807704 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807706 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807708 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807711 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807713 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807716 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807719 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807723 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807726 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807728 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807731 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807733 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807737 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807739 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.811784 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807742 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807744 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807747 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807749 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807752 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807755 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807758 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807760 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807763 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807766 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807768 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807771 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807773 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807776 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.807780 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:19:05.812275 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807895 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807899 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807902 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807906 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807909 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807911 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807932 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807937 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807940 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807942 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807945 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807949 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807951 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807953 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807956 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807959 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807961 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807964 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807967 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807969 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:19:05.812632 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807971 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807974 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807976 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807984 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807987 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807990 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807992 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807995 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.807998 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808000 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808002 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808004 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808007 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808010 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808012 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808014 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808017 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808019 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808021 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:19:05.813126 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808024 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808028 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808031 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808034 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808036 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808039 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808041 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808044 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808047 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808049 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808052 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808054 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808057 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808059 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808062 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808064 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808067 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808070 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808073 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808075 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:19:05.813601 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808077 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808080 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808082 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808085 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808087 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808090 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808092 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808095 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808097 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808099 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808102 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808104 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808107 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808109 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808112 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808114 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808117 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808119 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808121 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808133 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:19:05.814108 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808137 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808140 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808143 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808147 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808150 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808153 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808156 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808158 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808161 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808165 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808168 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808171 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808174 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808176 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808179 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808181 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808184 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808186 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808189 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:19:05.814586 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:05.808191 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.808196 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.808313 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.810722 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.812541 2577 server.go:1019] "Starting client certificate rotation" Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.812643 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:19:05.815054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.812683 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:19:05.815295 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.815281 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:19:05.817536 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.816455 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:19:05.829014 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.828993 2577 log.go:25] "Validated CRI v1 runtime API" Jul 09 16:19:05.831356 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.831339 2577 log.go:25] "Validated CRI v1 image API" Jul 09 16:19:05.832043 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.832032 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 16:19:05.834777 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.834753 2577 fs.go:135] Filesystem UUIDs: map[0821f99e-86dc-4087-aa61-b88b3966f016:/dev/nvme0n1p3 2daa394c-83e9-4984-8060-cf1fa355504f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 09 16:19:05.834850 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.834777 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 16:19:05.840398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.840284 2577 manager.go:216] Machine: {Timestamp:2026-07-09 16:19:05.838768383 +0000 UTC m=+0.363092706 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099009 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2338fd9155b423b3c657346833b2e3 SystemUUID:ec2338fd-9155-b423-b3c6-57346833b2e3 BootID:daf94a71-bfae-45bf-a044-3fe7b631e39d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:37:9c:69:a9:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:37:9c:69:a9:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:5e:31:9c:3d:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 16:19:05.840398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.840391 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 16:19:05.840514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.840471 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 16:19:05.843966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.843932 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 16:19:05.844115 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.843970 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 16:19:05.844883 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.844872 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 16:19:05.844933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.844885 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 16:19:05.844933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.844893 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:19:05.845813 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.845801 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:19:05.847524 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.847507 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 16:19:05.849407 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.849389 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:19:05.852260 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852247 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:19:05.852374 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852357 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 16:19:05.852408 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852392 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 09 16:19:05.852408 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852403 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 16:19:05.852469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852418 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 16:19:05.852469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852427 2577 kubelet.go:398] "Adding apiserver pod source" Jul 09 16:19:05.852469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852436 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 16:19:05.852800 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852787 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:19:05.852838 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.852806 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:19:05.855083 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.855066 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 16:19:05.857680 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857665 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 16:19:05.857751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857697 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 16:19:05.857859 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857849 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 16:19:05.857890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857864 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 16:19:05.857890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857871 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 16:19:05.857890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857876 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 16:19:05.857890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857883 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 16:19:05.857890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.857888 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 16:19:05.858489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858478 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 16:19:05.858489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858488 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 16:19:05.858542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858495 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 16:19:05.858542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858500 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 16:19:05.858542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858515 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 16:19:05.858542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.858523 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 16:19:05.859334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.859324 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 16:19:05.859334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.859333 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 16:19:05.862508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.862494 2577 server.go:1268] "Started kubelet" Jul 09 16:19:05.862646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.862579 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 16:19:05.862684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.862595 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 16:19:05.862684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.862674 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 16:19:05.863414 ip-10-0-133-162 systemd[1]: Started Kubernetes Kubelet. Jul 09 16:19:05.867449 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.867413 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 16:19:05.868790 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.868767 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 16:19:05.869603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.869583 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 09 16:19:05.871238 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.871216 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 16:19:05.871321 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.871261 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 16:19:05.871387 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.871372 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 16:19:05.872632 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.871529 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-162.ec2.internal.18c0ab6f0b90a77c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-162.ec2.internal,UID:ip-10-0-133-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-162.ec2.internal,},FirstTimestamp:2026-07-09 16:19:05.862465404 +0000 UTC m=+0.386789727,LastTimestamp:2026-07-09 16:19:05.862465404 +0000 UTC m=+0.386789727,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-162.ec2.internal,}" Jul 09 16:19:05.875958 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.875942 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 16:19:05.876047 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.875967 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 16:19:05.876047 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.876037 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:05.876972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.876954 2577 factory.go:55] Registering systemd factory Jul 09 16:19:05.877060 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.876976 2577 factory.go:223] Registration of the systemd container factory successfully Jul 09 16:19:05.877060 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877034 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 16:19:05.877060 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877043 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 16:19:05.877060 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877048 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 16:19:05.877233 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877224 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 09 16:19:05.877233 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877231 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 09 16:19:05.877338 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877320 2577 factory.go:153] Registering CRI-O factory Jul 09 16:19:05.877338 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877338 2577 factory.go:223] Registration of the crio container factory successfully Jul 09 16:19:05.877484 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877412 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 16:19:05.877484 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877438 2577 factory.go:103] Registering Raw factory Jul 09 16:19:05.877484 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877453 2577 manager.go:1196] Started watching for new ooms in manager Jul 09 16:19:05.877872 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.877857 2577 manager.go:349] Starting recovery of all containers Jul 09 16:19:05.886225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.886057 2577 manager.go:354] Recovery completed Jul 09 16:19:05.888663 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.888631 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 16:19:05.888746 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.888664 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 16:19:05.890876 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.890863 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:05.892890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.892872 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:05.893046 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.892908 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:05.893105 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.893073 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:05.893620 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.893606 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 16:19:05.893620 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.893620 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 16:19:05.893700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.893629 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:05.893700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.893635 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:19:05.894622 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.894607 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2n2rd" Jul 09 16:19:05.895427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.895412 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:05.895477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.895438 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:05.895477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.895448 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:05.895991 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.895977 2577 policy_none.go:49] "None policy: Start" Jul 09 16:19:05.896033 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.895999 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 16:19:05.896033 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.896010 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 16:19:05.897982 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.897971 2577 policy_none.go:47] "Start" Jul 09 16:19:05.903962 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.903845 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-162.ec2.internal.18c0ab6f0d60e8b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-162.ec2.internal,UID:ip-10-0-133-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-162.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-162.ec2.internal,},FirstTimestamp:2026-07-09 16:19:05.892890801 +0000 UTC m=+0.417215134,LastTimestamp:2026-07-09 16:19:05.892890801 +0000 UTC m=+0.417215134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-162.ec2.internal,}" Jul 09 16:19:05.905289 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.905268 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2n2rd" Jul 09 16:19:05.907663 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.907597 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-162.ec2.internal.18c0ab6f0d63792f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-162.ec2.internal,UID:ip-10-0-133-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-162.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-162.ec2.internal,},FirstTimestamp:2026-07-09 16:19:05.893058863 +0000 UTC m=+0.417383186,LastTimestamp:2026-07-09 16:19:05.893058863 +0000 UTC m=+0.417383186,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-162.ec2.internal,}" Jul 09 16:19:05.937536 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.937517 2577 manager.go:337] "Starting Device Plugin manager" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.937560 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.937570 2577 server.go:87] "Starting device plugin registration server" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.937842 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.937853 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.937971 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.938069 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.938078 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.938623 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 16:19:05.958630 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.938673 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:05.985070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.985040 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 16:19:05.986461 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.986440 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 16:19:05.986555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.986480 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 16:19:05.986555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.986504 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 16:19:05.986555 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:05.986536 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 16:19:05.989419 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:05.989395 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:06.038339 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.038257 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:06.039420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.039403 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:06.039513 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.039432 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:06.039513 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.039443 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:06.039513 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.039469 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.050401 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.050382 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.050483 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.050409 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-162.ec2.internal\": node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.083472 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.083441 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.087439 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.087416 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal"] Jul 09 16:19:06.087530 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.087499 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:06.089053 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.089034 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:06.089136 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.089069 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:06.089136 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.089083 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:06.091538 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.091525 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:06.091680 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.091653 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.091717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.091694 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:06.092262 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092248 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:06.092346 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092270 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:06.092346 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092281 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:06.092346 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092313 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:06.092488 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092383 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:06.092488 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.092401 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:06.094546 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.094531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.094607 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.094555 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:19:06.095189 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.095164 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:19:06.095189 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.095191 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:19:06.095314 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.095206 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:19:06.127659 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.127629 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-162.ec2.internal\" not found" node="ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.132065 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.132047 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-162.ec2.internal\" not found" node="ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.178543 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.178511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d48bc14c4e35da0c778b71f5201769c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"d48bc14c4e35da0c778b71f5201769c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.178543 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.178547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.178700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.178564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.183910 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.183890 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.279310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d48bc14c4e35da0c778b71f5201769c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"d48bc14c4e35da0c778b71f5201769c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.279310 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.279456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.279456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.279456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d48bc14c4e35da0c778b71f5201769c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-162.ec2.internal\" (UID: \"d48bc14c4e35da0c778b71f5201769c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.279456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.279389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c4ccb351969bc148f0a134c7c59730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal\" (UID: \"28c4ccb351969bc148f0a134c7c59730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.284404 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.284383 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.385306 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.385233 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.429438 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.429402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.435525 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.435504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.486325 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.486287 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.586877 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.586830 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.687451 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.687376 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.728214 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.728188 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:06.788486 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.788455 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.811649 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.811617 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 16:19:06.811794 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.811765 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:19:06.811835 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.811799 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:19:06.888788 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:06.888768 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-162.ec2.internal\" not found" Jul 09 16:19:06.910311 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.910269 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 16:14:06 +0000 UTC" deadline="2027-12-09 00:20:21.862759648 +0000 UTC" Jul 09 16:19:06.910311 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.910301 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12416h1m14.952462507s" Jul 09 16:19:06.931189 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.931168 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:06.976365 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.976322 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.993803 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.993782 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:19:06.995313 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:06.995293 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" Jul 09 16:19:06.995638 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:06.995609 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c4ccb351969bc148f0a134c7c59730.slice/crio-fa752d4ca6c0752a8045e7f25e609447012529125cfe1e317707764f961e90fe WatchSource:0}: Error finding container fa752d4ca6c0752a8045e7f25e609447012529125cfe1e317707764f961e90fe: Status 404 returned error can't find the container with id fa752d4ca6c0752a8045e7f25e609447012529125cfe1e317707764f961e90fe Jul 09 16:19:07.000073 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.000059 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:19:07.005448 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.005428 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:19:07.288315 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.288249 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:07.853436 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.853393 2577 apiserver.go:52] "Watching apiserver" Jul 09 16:19:07.863685 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.863650 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 16:19:07.865365 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.865333 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-dzjz6","openshift-network-diagnostics/network-check-target-rd9v8","kube-system/konnectivity-agent-jcrhw","kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gvkmx","openshift-dns/node-resolver-g8snr","openshift-image-registry/node-ca-zxlmb","openshift-multus/multus-qcpwx","openshift-network-operator/iptables-alerter-rzs7c","openshift-ovn-kubernetes/ovnkube-node-8482k","kube-system/global-pull-secret-syncer-kr65g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-j4j5t"] Jul 09 16:19:07.868160 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.868136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.870597 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.870571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:07.870710 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.870648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:07.871600 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.871578 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873084 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873146 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873204 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873157 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873535 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 16:19:07.873744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.873580 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cgwdj\"" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.875509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.875599 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.876164 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.876175 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mp5mz\"" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.876365 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 16:19:07.877012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.876419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 16:19:07.878687 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.878661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.881684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.881312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:07.881684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.881375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.881810 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.881692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.881964 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.881942 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r2nlj\"" Jul 09 16:19:07.883910 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.883889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.884978 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.884844 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.884978 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.884900 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vqtwq\"" Jul 09 16:19:07.885518 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.885496 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.886809 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.886884 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dz6mk\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.887114 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.887136 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.887166 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.887186 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 16:19:07.888024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-netns\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-systemd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-log-socket\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888129 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4cj\" (UniqueName: \"kubernetes.io/projected/b74aa39e-e278-4643-b285-59f08edc7821-kube-api-access-4j4cj\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-modprobe-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysconfig\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-systemd-units\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-node-log\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-netd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-config\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888458 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-etc-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-agent-certs\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888460 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-run\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-tmp\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-kubelet\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.888955 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.888743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-bin\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.889817 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.889797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:07.889949 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.889896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-slash\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890015 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.889944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890352 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9xq\" (UniqueName: \"kubernetes.io/projected/a659e30c-c220-4d77-8159-c4d452ea8e95-kube-api-access-cz9xq\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:07.890429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-var-lib-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-konnectivity-ca\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-kubernetes\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-systemd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-sys\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-ovn\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74aa39e-e278-4643-b285-59f08edc7821-ovn-node-metrics-cert\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-lib-modules\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-var-lib-kubelet\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-env-overrides\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-script-lib\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-conf\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-host\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-tuned\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.890855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.890778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7pd\" (UniqueName: \"kubernetes.io/projected/e5dd071b-7461-4178-bc7c-206ebbb7dd01-kube-api-access-zl7pd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.891392 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.891372 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 16:19:07.891462 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.891389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.891525 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.891507 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ffv6z\"" Jul 09 16:19:07.894246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.894225 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.894329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.894299 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rlfsb\"" Jul 09 16:19:07.894329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.894317 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 16:19:07.894400 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.894235 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.895529 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.895510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:07.895621 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.895590 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:07.898018 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.897990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:07.901242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.900522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:07.901242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.901010 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.901242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.901010 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.901242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.901049 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 16:19:07.901242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.901039 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wwf2\"" Jul 09 16:19:07.903570 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.903552 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 16:19:07.903800 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.903751 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 16:19:07.905789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.905704 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zds8c\"" Jul 09 16:19:07.906053 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.906029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 16:19:07.913505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.913481 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p85kf" Jul 09 16:19:07.922091 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.922066 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p85kf" Jul 09 16:19:07.978734 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.978701 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 16:19:07.990909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-bin\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:07.991074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-multus\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.991074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.991074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"28c4ccb351969bc148f0a134c7c59730","Type":"ContainerStarted","Data":"fa752d4ca6c0752a8045e7f25e609447012529125cfe1e317707764f961e90fe"} Jul 09 16:19:07.991074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.990999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-cni-binary-copy\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-netns\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-bin\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj95f\" (UniqueName: \"kubernetes.io/projected/3d10c6d0-5fa2-469f-87f1-913829110c26-kube-api-access-dj95f\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skl59\" (UniqueName: \"kubernetes.io/projected/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-kube-api-access-skl59\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-konnectivity-ca\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-sys\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-var-lib-kubelet\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-os-release\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-bin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.991377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-sys\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-var-lib-kubelet\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d10c6d0-5fa2-469f-87f1-913829110c26-host\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-ovn\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74aa39e-e278-4643-b285-59f08edc7821-ovn-node-metrics-cert\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991606 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-lib-modules\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-ovn\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzv9p\" (UniqueName: \"kubernetes.io/projected/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-kube-api-access-lzv9p\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-conf\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.991808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-tuned\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7pd\" (UniqueName: \"kubernetes.io/projected/e5dd071b-7461-4178-bc7c-206ebbb7dd01-kube-api-access-zl7pd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-cnibin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf2r\" (UniqueName: \"kubernetes.io/projected/d78730c6-cf40-490f-8c55-807f6db8cbf7-kube-api-access-9jf2r\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-dbus\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-netns\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4cj\" (UniqueName: \"kubernetes.io/projected/b74aa39e-e278-4643-b285-59f08edc7821-kube-api-access-4j4cj\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-tmp-dir\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-hostroot\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-etc-kubernetes\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.991992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-iptables-alerter-script\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-node-log\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992029 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-run-netns\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-netd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-lib-modules\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-config\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-kubelet\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.992425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-cni-netd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-conf\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-socket-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992454 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzn8\" (UniqueName: \"kubernetes.io/projected/c379949a-75ff-4eb7-a605-f6f4a96509f2-kube-api-access-rtzn8\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-node-log\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-konnectivity-ca\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-etc-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992531 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.992637 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-tmp\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-etc-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-hosts-file\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d10c6d0-5fa2-469f-87f1-913829110c26-serviceca\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-system-cni-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.992989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.993048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:08.493009996 +0000 UTC m=+3.017334332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:07.993332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-kubelet\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-conf-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-host-slash\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-kubelet\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-slash\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994007 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-k8s-cni-cncf-io\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-os-release\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9xq\" (UniqueName: \"kubernetes.io/projected/a659e30c-c220-4d77-8159-c4d452ea8e95-kube-api-access-cz9xq\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-var-lib-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-kubernetes\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.994286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-systemd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.994562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdfz\" (UniqueName: \"kubernetes.io/projected/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-kube-api-access-vxdfz\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.994562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-device-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.994562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-slash\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.993236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-config\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.994892 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-var-lib-openvswitch\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995020 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.994964 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-kubernetes\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995142 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995117 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysctl-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995207 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-systemd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995356 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-kubelet-config\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:07.995429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-socket-dir-parent\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.995429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995529 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-env-overrides\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995529 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-script-lib\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-host\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-systemd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-log-socket\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.995781 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-modprobe-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995781 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysconfig\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.995884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-daemon-config\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.995884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cnibin\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.996005 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.996005 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.995913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-sys-fs\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-systemd-units\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-env-overrides\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-system-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996100 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-sysconfig\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.996123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-multus-certs\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-tuned\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-agent-certs\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-run\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74aa39e-e278-4643-b285-59f08edc7821-ovn-node-metrics-cert\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-run\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.996428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-run-systemd\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-registration-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b74aa39e-e278-4643-b285-59f08edc7821-ovnkube-script-lib\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-host\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-log-socket\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b74aa39e-e278-4643-b285-59f08edc7821-systemd-units\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:07.996820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5dd071b-7461-4178-bc7c-206ebbb7dd01-etc-modprobe-d\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.997317 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.996913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5dd071b-7461-4178-bc7c-206ebbb7dd01-tmp\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:07.998116 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.998094 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:07.998286 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.998125 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:07.998286 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.998149 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:07.998286 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:07.998229 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:08.498199138 +0000 UTC m=+3.022523452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:07.999494 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:07.999472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8c7abf-3301-4730-a0f4-35d9c5bffe69-agent-certs\") pod \"konnectivity-agent-jcrhw\" (UID: \"0b8c7abf-3301-4730-a0f4-35d9c5bffe69\") " pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:08.001384 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.001351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7pd\" (UniqueName: \"kubernetes.io/projected/e5dd071b-7461-4178-bc7c-206ebbb7dd01-kube-api-access-zl7pd\") pod \"tuned-gvkmx\" (UID: \"e5dd071b-7461-4178-bc7c-206ebbb7dd01\") " pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:08.001477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.001457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4cj\" (UniqueName: \"kubernetes.io/projected/b74aa39e-e278-4643-b285-59f08edc7821-kube-api-access-4j4cj\") pod \"ovnkube-node-8482k\" (UID: \"b74aa39e-e278-4643-b285-59f08edc7821\") " pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:08.007469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.007445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9xq\" (UniqueName: \"kubernetes.io/projected/a659e30c-c220-4d77-8159-c4d452ea8e95-kube-api-access-cz9xq\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:08.096749 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-host-slash\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.096749 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-k8s-cni-cncf-io\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-os-release\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdfz\" (UniqueName: \"kubernetes.io/projected/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-kube-api-access-vxdfz\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-host-slash\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-k8s-cni-cncf-io\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-os-release\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-device-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-kubelet-config\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.096991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-socket-dir-parent\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-device-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.097041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-daemon-config\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-kubelet-config\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cnibin\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-socket-dir-parent\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cnibin\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-sys-fs\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-system-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-sys-fs\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-multus-certs\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-multus-certs\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-registration-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.097442 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:08.097542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-multus\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-cni-binary-copy\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-netns\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.097559 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:08.597538432 +0000 UTC m=+3.121862751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-multus\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097581 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-run-netns\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-registration-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-system-cni-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj95f\" (UniqueName: \"kubernetes.io/projected/3d10c6d0-5fa2-469f-87f1-913829110c26-kube-api-access-dj95f\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skl59\" (UniqueName: \"kubernetes.io/projected/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-kube-api-access-skl59\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-os-release\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-bin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d10c6d0-5fa2-469f-87f1-913829110c26-host\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzv9p\" (UniqueName: \"kubernetes.io/projected/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-kube-api-access-lzv9p\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.098520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-cnibin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf2r\" (UniqueName: \"kubernetes.io/projected/d78730c6-cf40-490f-8c55-807f6db8cbf7-kube-api-access-9jf2r\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-dbus\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-tmp-dir\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-hostroot\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-etc-kubernetes\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-iptables-alerter-script\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-daemon-config\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d78730c6-cf40-490f-8c55-807f6db8cbf7-cni-binary-copy\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-kubelet\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-kubelet\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-host-var-lib-cni-bin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-os-release\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.097791 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-socket-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.099303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d10c6d0-5fa2-469f-87f1-913829110c26-host\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-hostroot\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-etc-kubernetes\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-socket-dir\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098450 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19baab78-2289-4c52-a951-febcda3d048a-dbus\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzn8\" (UniqueName: \"kubernetes.io/projected/c379949a-75ff-4eb7-a605-f6f4a96509f2-kube-api-access-rtzn8\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-tmp-dir\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-hosts-file\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098572 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-cnibin\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d10c6d0-5fa2-469f-87f1-913829110c26-serviceca\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-system-cni-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-iptables-alerter-script\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-hosts-file\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-conf-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-system-cni-dir\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098784 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c379949a-75ff-4eb7-a605-f6f4a96509f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d78730c6-cf40-490f-8c55-807f6db8cbf7-multus-conf-dir\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.099900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.098974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d10c6d0-5fa2-469f-87f1-913829110c26-serviceca\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.114285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.113651 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf2r\" (UniqueName: \"kubernetes.io/projected/d78730c6-cf40-490f-8c55-807f6db8cbf7-kube-api-access-9jf2r\") pod \"multus-qcpwx\" (UID: \"d78730c6-cf40-490f-8c55-807f6db8cbf7\") " pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.114285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.114044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdfz\" (UniqueName: \"kubernetes.io/projected/7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88-kube-api-access-vxdfz\") pod \"multus-additional-cni-plugins-j4j5t\" (UID: \"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88\") " pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.114285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.114176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj95f\" (UniqueName: \"kubernetes.io/projected/3d10c6d0-5fa2-469f-87f1-913829110c26-kube-api-access-dj95f\") pod \"node-ca-zxlmb\" (UID: \"3d10c6d0-5fa2-469f-87f1-913829110c26\") " pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.115159 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.114912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzv9p\" (UniqueName: \"kubernetes.io/projected/df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87-kube-api-access-lzv9p\") pod \"node-resolver-g8snr\" (UID: \"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87\") " pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.115159 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.115115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skl59\" (UniqueName: \"kubernetes.io/projected/d1ba6608-a9b4-42b5-b6df-68cecb3e7715-kube-api-access-skl59\") pod \"iptables-alerter-rzs7c\" (UID: \"d1ba6608-a9b4-42b5-b6df-68cecb3e7715\") " pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.115483 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.115462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzn8\" (UniqueName: \"kubernetes.io/projected/c379949a-75ff-4eb7-a605-f6f4a96509f2-kube-api-access-rtzn8\") pod \"aws-ebs-csi-driver-node-k945k\" (UID: \"c379949a-75ff-4eb7-a605-f6f4a96509f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.137407 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.137379 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:08.181862 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.181812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:08.188779 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.188755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:08.193729 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.193700 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74aa39e_e278_4643_b285_59f08edc7821.slice/crio-c19fa872fb20afeba449f42a7828712b145edf9bdad7bf52585cec0b1fe522ea WatchSource:0}: Error finding container c19fa872fb20afeba449f42a7828712b145edf9bdad7bf52585cec0b1fe522ea: Status 404 returned error can't find the container with id c19fa872fb20afeba449f42a7828712b145edf9bdad7bf52585cec0b1fe522ea Jul 09 16:19:08.199192 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.199165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" Jul 09 16:19:08.199264 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.199232 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8c7abf_3301_4730_a0f4_35d9c5bffe69.slice/crio-d521d4cb0708de0868aecaa4356a85a1f8260c321d9012dc7f66ce15664d4d2a WatchSource:0}: Error finding container d521d4cb0708de0868aecaa4356a85a1f8260c321d9012dc7f66ce15664d4d2a: Status 404 returned error can't find the container with id d521d4cb0708de0868aecaa4356a85a1f8260c321d9012dc7f66ce15664d4d2a Jul 09 16:19:08.205145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.205127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8snr" Jul 09 16:19:08.209880 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.209856 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd071b_7461_4178_bc7c_206ebbb7dd01.slice/crio-c7528712d6290f6252213cdf1d3ab92db39b92c3656599447a4f8138107b72fd WatchSource:0}: Error finding container c7528712d6290f6252213cdf1d3ab92db39b92c3656599447a4f8138107b72fd: Status 404 returned error can't find the container with id c7528712d6290f6252213cdf1d3ab92db39b92c3656599447a4f8138107b72fd Jul 09 16:19:08.213228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.213207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" Jul 09 16:19:08.216059 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.214960 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf15f9e5_7f65_49c2_81e4_9a5ed9bf4c87.slice/crio-986b8db347692952de19c17942e10d1b53935708d1a88917da18bc08374294f9 WatchSource:0}: Error finding container 986b8db347692952de19c17942e10d1b53935708d1a88917da18bc08374294f9: Status 404 returned error can't find the container with id 986b8db347692952de19c17942e10d1b53935708d1a88917da18bc08374294f9 Jul 09 16:19:08.219757 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.219741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qcpwx" Jul 09 16:19:08.223044 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.222929 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbd70a8_4d2e_4f6f_a7ed_4953b91a4a88.slice/crio-3a49d3b01f7c8ef1be9347e31b322c39a15e2d81b18be30b5e582a58526c521f WatchSource:0}: Error finding container 3a49d3b01f7c8ef1be9347e31b322c39a15e2d81b18be30b5e582a58526c521f: Status 404 returned error can't find the container with id 3a49d3b01f7c8ef1be9347e31b322c39a15e2d81b18be30b5e582a58526c521f Jul 09 16:19:08.227967 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.227947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" Jul 09 16:19:08.229603 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.229578 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78730c6_cf40_490f_8c55_807f6db8cbf7.slice/crio-81bf68b15ad59a4e19ce79f3d4e36667b53a7e274d52e61d23b11dffeda32890 WatchSource:0}: Error finding container 81bf68b15ad59a4e19ce79f3d4e36667b53a7e274d52e61d23b11dffeda32890: Status 404 returned error can't find the container with id 81bf68b15ad59a4e19ce79f3d4e36667b53a7e274d52e61d23b11dffeda32890 Jul 09 16:19:08.234453 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.234437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzs7c" Jul 09 16:19:08.238637 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.238448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxlmb" Jul 09 16:19:08.238829 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.238802 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc379949a_75ff_4eb7_a605_f6f4a96509f2.slice/crio-2d392f11244a4067e9c6c724e632d30bed502b4e282d67843f0f3f7195387db3 WatchSource:0}: Error finding container 2d392f11244a4067e9c6c724e632d30bed502b4e282d67843f0f3f7195387db3: Status 404 returned error can't find the container with id 2d392f11244a4067e9c6c724e632d30bed502b4e282d67843f0f3f7195387db3 Jul 09 16:19:08.245939 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.245912 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ba6608_a9b4_42b5_b6df_68cecb3e7715.slice/crio-efc199099546f140b6cbf19da84e7f60eeaabe361f87e9987fac3ef7860ac4c4 WatchSource:0}: Error finding container efc199099546f140b6cbf19da84e7f60eeaabe361f87e9987fac3ef7860ac4c4: Status 404 returned error can't find the container with id efc199099546f140b6cbf19da84e7f60eeaabe361f87e9987fac3ef7860ac4c4 Jul 09 16:19:08.249883 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:08.249859 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d10c6d0_5fa2_469f_87f1_913829110c26.slice/crio-b1c70353147bf8f80db676c596125772f9ce984a826ab829b919df02f923ab01 WatchSource:0}: Error finding container b1c70353147bf8f80db676c596125772f9ce984a826ab829b919df02f923ab01: Status 404 returned error can't find the container with id b1c70353147bf8f80db676c596125772f9ce984a826ab829b919df02f923ab01 Jul 09 16:19:08.501931 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.501862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:08.502119 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.501974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:08.502204 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502141 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:08.502244 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502206 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:09.502184892 +0000 UTC m=+4.026509204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:08.502487 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502457 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:08.502487 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502485 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:08.502661 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502498 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:08.502661 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.502583 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:09.502543411 +0000 UTC m=+4.026867734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:08.602988 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.602933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:08.603217 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.603197 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:08.603299 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.603269 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:09.603250261 +0000 UTC m=+4.127574573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:08.926085 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.925970 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:14:08 +0000 UTC" deadline="2028-01-19 00:55:34.81622952 +0000 UTC" Jul 09 16:19:08.926085 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.926035 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13400h36m25.890199017s" Jul 09 16:19:08.988232 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.987355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:08.988232 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.987497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:08.988232 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:08.987960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:08.988232 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:08.988095 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:09.004188 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.003966 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" event={"ID":"c379949a-75ff-4eb7-a605-f6f4a96509f2","Type":"ContainerStarted","Data":"2d392f11244a4067e9c6c724e632d30bed502b4e282d67843f0f3f7195387db3"} Jul 09 16:19:09.010730 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.010620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerStarted","Data":"3a49d3b01f7c8ef1be9347e31b322c39a15e2d81b18be30b5e582a58526c521f"} Jul 09 16:19:09.016202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.016109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" event={"ID":"e5dd071b-7461-4178-bc7c-206ebbb7dd01","Type":"ContainerStarted","Data":"c7528712d6290f6252213cdf1d3ab92db39b92c3656599447a4f8138107b72fd"} Jul 09 16:19:09.028215 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.027863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jcrhw" event={"ID":"0b8c7abf-3301-4730-a0f4-35d9c5bffe69","Type":"ContainerStarted","Data":"d521d4cb0708de0868aecaa4356a85a1f8260c321d9012dc7f66ce15664d4d2a"} Jul 09 16:19:09.030871 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.030566 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" event={"ID":"d48bc14c4e35da0c778b71f5201769c5","Type":"ContainerStarted","Data":"35c7ea90b3ea64c734f22dbc54cbb54c45ff4518355a473329d55e1f8c587538"} Jul 09 16:19:09.043111 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.043044 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qcpwx" event={"ID":"d78730c6-cf40-490f-8c55-807f6db8cbf7","Type":"ContainerStarted","Data":"81bf68b15ad59a4e19ce79f3d4e36667b53a7e274d52e61d23b11dffeda32890"} Jul 09 16:19:09.045418 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.045376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8snr" event={"ID":"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87","Type":"ContainerStarted","Data":"986b8db347692952de19c17942e10d1b53935708d1a88917da18bc08374294f9"} Jul 09 16:19:09.055127 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.054923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"c19fa872fb20afeba449f42a7828712b145edf9bdad7bf52585cec0b1fe522ea"} Jul 09 16:19:09.063367 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.063330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxlmb" event={"ID":"3d10c6d0-5fa2-469f-87f1-913829110c26","Type":"ContainerStarted","Data":"b1c70353147bf8f80db676c596125772f9ce984a826ab829b919df02f923ab01"} Jul 09 16:19:09.089327 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.089183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzs7c" event={"ID":"d1ba6608-a9b4-42b5-b6df-68cecb3e7715","Type":"ContainerStarted","Data":"efc199099546f140b6cbf19da84e7f60eeaabe361f87e9987fac3ef7860ac4c4"} Jul 09 16:19:09.104608 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.104575 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:19:09.509746 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.509704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:09.509917 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.509796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:09.510015 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.509979 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:09.510015 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.510012 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:09.510140 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.510037 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:09.510140 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.510101 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:11.510080865 +0000 UTC m=+6.034405195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:09.510559 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.510535 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:09.510677 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.510597 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:11.510580706 +0000 UTC m=+6.034905019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:09.610370 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.610330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:09.610565 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.610520 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:09.610631 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.610591 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:11.610570626 +0000 UTC m=+6.134894941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:09.927013 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.926954 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:14:08 +0000 UTC" deadline="2028-04-23 22:42:18.564479934 +0000 UTC" Jul 09 16:19:09.927013 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.927001 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15702h23m8.63748507s" Jul 09 16:19:09.988171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:09.988130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:09.988376 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:09.988277 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:10.121947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:10.121909 2577 generic.go:356] "Generic (PLEG): container finished" podID="28c4ccb351969bc148f0a134c7c59730" containerID="a88b65c92f90002359d86d39fbb44266bf28265bf8b4a17c4354f10e143d5df7" exitCode=0 Jul 09 16:19:10.122143 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:10.121966 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"28c4ccb351969bc148f0a134c7c59730","Type":"ContainerDied","Data":"a88b65c92f90002359d86d39fbb44266bf28265bf8b4a17c4354f10e143d5df7"} Jul 09 16:19:10.986833 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:10.986797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:10.987633 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:10.986974 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:10.987633 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:10.987383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:10.987633 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:10.987486 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:11.144369 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.144324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" event={"ID":"28c4ccb351969bc148f0a134c7c59730","Type":"ContainerStarted","Data":"b563c203dad7ea6b98373e878ad670ff0ca1826b14e12d5381ac44a38d0cdcbd"} Jul 09 16:19:11.170676 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.170604 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-162.ec2.internal" podStartSLOduration=4.170578631 podStartE2EDuration="4.170578631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:11.169372975 +0000 UTC m=+5.693697313" watchObservedRunningTime="2026-07-09 16:19:11.170578631 +0000 UTC m=+5.694902969" Jul 09 16:19:11.527297 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.527248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:11.527540 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.527343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:11.527540 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.527510 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:11.527540 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.527528 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:11.527540 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.527542 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:11.527763 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.527600 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:15.527580509 +0000 UTC m=+10.051904824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:11.528154 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.528133 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:11.528246 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.528191 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:15.528174801 +0000 UTC m=+10.052499118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:11.628118 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.628079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:11.628306 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.628282 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:11.628387 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.628354 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:15.628336549 +0000 UTC m=+10.152660867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:11.989909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:11.989874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:11.990459 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:11.990035 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:12.987607 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:12.987566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:12.987874 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:12.987720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:12.988200 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:12.988176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:12.988305 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:12.988288 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:13.987354 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:13.986831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:13.987354 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:13.986987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:14.987104 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:14.987069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:14.987289 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:14.987074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:14.987289 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:14.987192 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:14.987425 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:14.987315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:15.560514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:15.560478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:15.560704 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:15.560546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:15.560704 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560674 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:15.560704 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560701 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:15.560858 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560715 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:15.560858 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560713 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:15.560858 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560764 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:23.560748836 +0000 UTC m=+18.085073150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:15.560858 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.560786 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:23.560775572 +0000 UTC m=+18.085099885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:15.662152 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:15.661620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:15.662152 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.661791 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:15.662152 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.661846 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:23.661828982 +0000 UTC m=+18.186153292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:15.990118 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:15.990084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:15.990577 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:15.990218 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:16.986784 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:16.986743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:16.986784 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:16.986767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:16.987067 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:16.986881 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:16.987131 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:16.987068 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:17.987228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:17.987191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:17.987597 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:17.987330 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:18.987196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:18.987160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:18.987364 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:18.987212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:18.987364 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:18.987300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:18.987767 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:18.987434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:19.986802 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:19.986769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:19.987008 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:19.986897 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:20.987617 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:20.987577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:20.988061 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:20.987691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:20.988061 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:20.987754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:20.988061 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:20.987868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:21.987533 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:21.987494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:21.987714 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:21.987636 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:22.987365 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:22.987326 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:22.987599 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:22.987334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:22.987599 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:22.987453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:22.987599 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:22.987548 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:23.625476 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:23.625436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:23.625505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625612 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625634 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625655 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625673 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:39.625676025 +0000 UTC m=+34.150000338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:23.625895 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.625717 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:39.625705586 +0000 UTC m=+34.150029897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:23.726218 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:23.726174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:23.726380 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.726343 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:23.726454 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.726403 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:19:39.726387352 +0000 UTC m=+34.250711664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:23.987361 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:23.987323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:23.987549 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:23.987467 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:24.987292 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:24.987254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:24.987718 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:24.987299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:24.987718 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:24.987413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:24.987718 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:24.987581 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:25.988328 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:25.988295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:25.988770 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:25.988417 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:26.987177 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:26.987137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:26.987358 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:26.987150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:26.987358 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:26.987281 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:26.987476 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:26.987408 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:27.987485 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:27.987052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:27.987485 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:27.987179 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:28.987159 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:28.987117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:28.987369 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:28.987137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:28.987369 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:28.987260 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:28.987369 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:28.987276 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:29.988119 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:29.987779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:29.988808 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:29.988210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:30.187249 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.187205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" event={"ID":"c379949a-75ff-4eb7-a605-f6f4a96509f2","Type":"ContainerStarted","Data":"e2463c9ca9703b9d729ae4b102504b30cb08816eff02fd8adb3a24ad61206044"} Jul 09 16:19:30.188868 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.188838 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="24dea92a0ab2858e5aeefdecc6877895d06d92de3ff323444ad7f69527dea143" exitCode=0 Jul 09 16:19:30.189026 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.188939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"24dea92a0ab2858e5aeefdecc6877895d06d92de3ff323444ad7f69527dea143"} Jul 09 16:19:30.190416 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.190361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" event={"ID":"e5dd071b-7461-4178-bc7c-206ebbb7dd01","Type":"ContainerStarted","Data":"7a41a446e8218e3dcd0692d1744ad0f1306909a0074a52207fdb6af0eca34e70"} Jul 09 16:19:30.191863 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.191842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jcrhw" event={"ID":"0b8c7abf-3301-4730-a0f4-35d9c5bffe69","Type":"ContainerStarted","Data":"e9fc4bee278eef13e9ea0ee1513ebd21d13d9c0ce3762c79957df0dd0542d069"} Jul 09 16:19:30.193229 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.193210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" event={"ID":"d48bc14c4e35da0c778b71f5201769c5","Type":"ContainerStarted","Data":"e480a806ec2312846afb4adbe572339390f007ea73e8397148a28407382fc169"} Jul 09 16:19:30.194661 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.194639 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qcpwx" event={"ID":"d78730c6-cf40-490f-8c55-807f6db8cbf7","Type":"ContainerStarted","Data":"8c22f695f30d980f15bde436f638631ec19db8a6175f0f17b856fd082b250c23"} Jul 09 16:19:30.196037 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.196016 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8snr" event={"ID":"df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87","Type":"ContainerStarted","Data":"a6b351ce1ad635b1afb8579c8d7f6f25615ff8eee9bde537b32fbd78f764ff11"} Jul 09 16:19:30.200387 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200365 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:19:30.200715 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200691 2577 generic.go:356] "Generic (PLEG): container finished" podID="b74aa39e-e278-4643-b285-59f08edc7821" containerID="cae4704b033568256447f2a62871341301565c4aec740d2586063bdb24464c27" exitCode=1 Jul 09 16:19:30.200792 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"c440a464d4a2b02971d5a7c0faf103ee77e2a7ace35e7dccea770359437d9bbc"} Jul 09 16:19:30.200849 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"7352cf4e79de71726899634bbeeb3670abb7b743b7c4ae13f8b444a36df62b19"} Jul 09 16:19:30.200903 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"94f6563905fc45165ecf4d1be43ff9da539b7c1bf383ea3846b9c09988cec129"} Jul 09 16:19:30.200982 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"77773dede61851682666c4822eb65e3a9eb68ce230dfccf7e3fea8a39306aa45"} Jul 09 16:19:30.200982 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.200964 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerDied","Data":"cae4704b033568256447f2a62871341301565c4aec740d2586063bdb24464c27"} Jul 09 16:19:30.201063 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.201014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"30bab0839252ac0ab348a1571b17c98a498139e1085e8b87462a1b245c2d22eb"} Jul 09 16:19:30.202110 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.202089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxlmb" event={"ID":"3d10c6d0-5fa2-469f-87f1-913829110c26","Type":"ContainerStarted","Data":"efafdcfe7a03ef91d10ecc770358745bae08a4cb448ace371d570b5bb439b236"} Jul 09 16:19:30.229001 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.228946 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zxlmb" podStartSLOduration=2.43221569 podStartE2EDuration="23.228932038s" totalImagesPullingTime="20.796716348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.228609481 +0000 UTC m=+24.752933812" watchObservedRunningTime="2026-07-09 16:19:30.228932038 +0000 UTC m=+24.753256361" Jul 09 16:19:30.242187 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.242099 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-162.ec2.internal" podStartSLOduration=23.242085019 podStartE2EDuration="23.242085019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.241876934 +0000 UTC m=+24.766201263" watchObservedRunningTime="2026-07-09 16:19:30.242085019 +0000 UTC m=+24.766409350" Jul 09 16:19:30.261438 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.261385 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jcrhw" podStartSLOduration=11.444649192 podStartE2EDuration="24.26137039s" totalImagesPullingTime="12.816721198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.261018908 +0000 UTC m=+24.785343240" watchObservedRunningTime="2026-07-09 16:19:30.26137039 +0000 UTC m=+24.785694756" Jul 09 16:19:30.278568 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.278524 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gvkmx" podStartSLOduration=3.441956369 podStartE2EDuration="24.278510055s" totalImagesPullingTime="20.836553686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.278422466 +0000 UTC m=+24.802746797" watchObservedRunningTime="2026-07-09 16:19:30.278510055 +0000 UTC m=+24.802834386" Jul 09 16:19:30.295596 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.295550 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qcpwx" podStartSLOduration=3.398578489 podStartE2EDuration="24.295534212s" totalImagesPullingTime="20.896955723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.295343547 +0000 UTC m=+24.819667879" watchObservedRunningTime="2026-07-09 16:19:30.295534212 +0000 UTC m=+24.819858543" Jul 09 16:19:30.314777 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.314735 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g8snr" podStartSLOduration=3.484512644 podStartE2EDuration="24.314718412s" totalImagesPullingTime="20.830205768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:30.314159915 +0000 UTC m=+24.838484247" watchObservedRunningTime="2026-07-09 16:19:30.314718412 +0000 UTC m=+24.839042744" Jul 09 16:19:30.987173 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.987140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:30.987456 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:30.987274 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:30.987456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:30.987296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:30.987456 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:30.987396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:31.205400 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.205362 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzs7c" event={"ID":"d1ba6608-a9b4-42b5-b6df-68cecb3e7715","Type":"ContainerStarted","Data":"19aa0d64789d02f3fb4bb8667dc9eb990cd8dd0ea04aff20e5b79f3b9511c05d"} Jul 09 16:19:31.223640 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.223590 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rzs7c" podStartSLOduration=3.4258155710000002 podStartE2EDuration="24.223571303s" totalImagesPullingTime="20.797755732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:31.223273611 +0000 UTC m=+25.747597942" watchObservedRunningTime="2026-07-09 16:19:31.223571303 +0000 UTC m=+25.747895634" Jul 09 16:19:31.358882 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.358856 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 16:19:31.836052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.835815 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:31.836630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.836607 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:31.950425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.950238 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:19:31.358873873Z","UUID":"67211912-2701-4f41-9b88-62db7cb2e590","Handler":null,"Name":"","Endpoint":""} Jul 09 16:19:31.953146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.952613 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 16:19:31.953146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.952659 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 16:19:31.988144 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:31.987642 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:31.988144 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:31.987784 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:32.210477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:32.210412 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:19:32.210856 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:32.210765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"2b58c965d1e1b3b7ca423c638a7d30091508cb56d57e11881d4ef492dc56eec0"} Jul 09 16:19:32.212374 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:32.212347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" event={"ID":"c379949a-75ff-4eb7-a605-f6f4a96509f2","Type":"ContainerStarted","Data":"9bfd9f4ee9d1a2deb6b2ee5974e59249203b2b6b11fe8714c40241a33b3be70f"} Jul 09 16:19:32.987444 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:32.987415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:32.987690 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:32.987455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:32.987690 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:32.987541 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:32.987690 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:32.987670 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:33.214823 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:33.214759 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 16:19:33.987246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:33.987145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:33.987401 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:33.987262 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:34.220533 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.220372 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:19:34.221215 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.220903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"8b11b08874abaebbd356028d1941e69af2714a6fa8f1d42ddd9df0a2db889f4c"} Jul 09 16:19:34.221215 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.221164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:34.221371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.221357 2577 scope.go:117] "RemoveContainer" containerID="cae4704b033568256447f2a62871341301565c4aec740d2586063bdb24464c27" Jul 09 16:19:34.223503 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.223478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" event={"ID":"c379949a-75ff-4eb7-a605-f6f4a96509f2","Type":"ContainerStarted","Data":"0dc1b394cdd9d556b5f407812d0af88eaf606ceb22164aeac268739d766ea01d"} Jul 09 16:19:34.238414 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.238391 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:34.293101 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.293027 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k945k" podStartSLOduration=7.334217583 podStartE2EDuration="28.293007365s" totalImagesPullingTime="20.958789782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:34.292547576 +0000 UTC m=+28.816871908" watchObservedRunningTime="2026-07-09 16:19:34.293007365 +0000 UTC m=+28.817331693" Jul 09 16:19:34.987679 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.987645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:34.987879 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:34.987788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:34.987879 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:34.987845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:34.988010 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:34.987989 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:35.232455 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.232429 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:19:35.232911 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.232779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" event={"ID":"b74aa39e-e278-4643-b285-59f08edc7821","Type":"ContainerStarted","Data":"802004161f49bda9d1e8013101f9fb128f04e7706ea9e603b2d8aaab060ff396"} Jul 09 16:19:35.233179 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.233154 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:35.233287 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.233255 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:35.253378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.253312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:19:35.264718 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.264667 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" podStartSLOduration=8.325654731 podStartE2EDuration="29.26464526s" totalImagesPullingTime="20.938990529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:35.264043964 +0000 UTC m=+29.788368296" watchObservedRunningTime="2026-07-09 16:19:35.26464526 +0000 UTC m=+29.788969625" Jul 09 16:19:35.527564 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.527479 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:35.527709 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.527628 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 16:19:35.528100 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.528071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jcrhw" Jul 09 16:19:35.988828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:35.988272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:35.988828 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:35.988400 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:36.227341 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.227310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzjz6"] Jul 09 16:19:36.227507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.227444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:36.227570 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:36.227552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:36.230522 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.230490 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kr65g"] Jul 09 16:19:36.231197 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.231116 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rd9v8"] Jul 09 16:19:36.231309 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.231213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:36.231373 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:36.231299 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:36.235565 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:36.235541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:36.236033 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:36.235713 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:37.239541 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:37.239371 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="74fbf0cd82e163d90c770b9a0fc6e49f4b2be1e7d07526d8468385d98f55ae37" exitCode=0 Jul 09 16:19:37.239541 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:37.239461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"74fbf0cd82e163d90c770b9a0fc6e49f4b2be1e7d07526d8468385d98f55ae37"} Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:37.989600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:37.989732 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:37.990210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:37.990320 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:37.990386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:37.990497 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:37.990458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:38.244970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:38.244911 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="ebdece10f844cdb2265c19f9c520476636c02e43b594f13bf7e830d6d8ceae80" exitCode=0 Jul 09 16:19:38.244970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:38.244959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"ebdece10f844cdb2265c19f9c520476636c02e43b594f13bf7e830d6d8ceae80"} Jul 09 16:19:39.642527 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.642490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.642541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642643 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642674 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642691 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs podName:a659e30c-c220-4d77-8159-c4d452ea8e95 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:11.642678541 +0000 UTC m=+66.167002851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs") pod "network-metrics-daemon-dzjz6" (UID: "a659e30c-c220-4d77-8159-c4d452ea8e95") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642700 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642713 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gglfg for pod openshift-network-diagnostics/network-check-target-rd9v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:39.643187 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.642765 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg podName:d739eb4d-2df5-4fd1-9bab-5ec49b834c84 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:11.642751381 +0000 UTC m=+66.167075708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gglfg" (UniqueName: "kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg") pod "network-check-target-rd9v8" (UID: "d739eb4d-2df5-4fd1-9bab-5ec49b834c84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:19:39.743500 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.743467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:39.743629 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.743600 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:39.743673 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.743666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret podName:19baab78-2289-4c52-a951-febcda3d048a nodeName:}" failed. No retries permitted until 2026-07-09 16:20:11.74365032 +0000 UTC m=+66.267974630 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret") pod "global-pull-secret-syncer-kr65g" (UID: "19baab78-2289-4c52-a951-febcda3d048a") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:19:39.986867 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.986838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:39.987067 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.986853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:39.987067 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.986973 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzjz6" podUID="a659e30c-c220-4d77-8159-c4d452ea8e95" Jul 09 16:19:39.987067 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:39.986852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:39.987067 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.987033 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rd9v8" podUID="d739eb4d-2df5-4fd1-9bab-5ec49b834c84" Jul 09 16:19:39.987217 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:39.987085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kr65g" podUID="19baab78-2289-4c52-a951-febcda3d048a" Jul 09 16:19:41.842184 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.842103 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-162.ec2.internal" event="NodeReady" Jul 09 16:19:41.842609 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.842267 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 16:19:41.878299 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.878267 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-kgtns"] Jul 09 16:19:41.881911 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.881877 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:19:41.882852 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.882632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.886049 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.885994 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpnk5\"" Jul 09 16:19:41.886171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886158 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml"] Jul 09 16:19:41.886253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 16:19:41.886321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.886577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886364 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.886577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886558 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.886745 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.886725 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 16:19:41.890061 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.890041 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6"] Jul 09 16:19:41.890168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.890151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" Jul 09 16:19:41.892422 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.892403 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rfjnp\"" Jul 09 16:19:41.893272 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 16:19:41.893272 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893264 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 16:19:41.893431 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893259 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 16:19:41.893582 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893562 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 16:19:41.893824 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893803 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f6db5c95b-l6zp4"] Jul 09 16:19:41.893935 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893863 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.893935 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.893893 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7g8qs\"" Jul 09 16:19:41.895013 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.894163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:41.898909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.897787 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl"] Jul 09 16:19:41.898909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.898156 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.898909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.898680 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.904578 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.899391 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:41.904578 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.901497 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 16:19:41.904578 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.903286 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 16:19:41.905863 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.905464 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 16:19:41.905863 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.905533 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.906775 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.906750 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 16:19:41.906775 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.906773 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5b7x6\"" Jul 09 16:19:41.907008 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.906991 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.907077 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907042 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 16:19:41.907128 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 16:19:41.907204 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907189 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vdvwq\"" Jul 09 16:19:41.907816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907328 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.907816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907388 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl"] Jul 09 16:19:41.907816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 16:19:41.907816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.907757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:41.911236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.910281 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 16:19:41.911236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.910674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rzfkh\"" Jul 09 16:19:41.911236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.910905 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.911236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.911059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 16:19:41.911236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.911071 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp"] Jul 09 16:19:41.911535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.911270 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.911535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.911360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:41.914113 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l7zxm\"" Jul 09 16:19:41.914371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914351 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.914477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.914544 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914525 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 16:19:41.914593 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914572 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 16:19:41.914785 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914768 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-6b2fz"] Jul 09 16:19:41.914859 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.914846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:41.917738 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.917715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.918045 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.918026 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.920768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.918365 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf"] Jul 09 16:19:41.920768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.918554 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-59jjp\"" Jul 09 16:19:41.920768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.918708 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 16:19:41.920768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.918958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.921467 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.921693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-njkbk\"" Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922235 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922443 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml"] Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922668 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp"] Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922700 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6"] Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl"] Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922741 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl"] Jul 09 16:19:41.922764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922758 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-kgtns"] Jul 09 16:19:41.923550 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.922778 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qnmkm"] Jul 09 16:19:41.923550 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.923501 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.924311 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.923802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" Jul 09 16:19:41.926394 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926374 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zfd8z"] Jul 09 16:19:41.926507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926382 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vdn2k\"" Jul 09 16:19:41.926507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926408 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.926507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:41.926747 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926731 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.926814 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.926737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 16:19:41.928860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.928843 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 16:19:41.929034 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.929018 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lp92w\"" Jul 09 16:19:41.929146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.929036 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 16:19:41.929146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.929140 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 16:19:41.930875 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.930858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:19:41.931012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.930883 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-6b2fz"] Jul 09 16:19:41.931012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.930896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnmkm"] Jul 09 16:19:41.931012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.930910 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6db5c95b-l6zp4"] Jul 09 16:19:41.931012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.930971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf"] Jul 09 16:19:41.931198 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.931024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:41.931246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.931214 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zfd8z"] Jul 09 16:19:41.933517 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.933492 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 16:19:41.933517 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.933514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-466pn\"" Jul 09 16:19:41.933755 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.933732 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 16:19:41.961048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkqm\" (UniqueName: \"kubernetes.io/projected/c71ed983-0744-4a12-9b54-fd0a633b24d9-kube-api-access-9lkqm\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961132 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961132 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-tmp\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961200 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-snapshots\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961200 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbcl2\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961200 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961284 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961284 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961345 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961345 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c71ed983-0744-4a12-9b54-fd0a633b24d9-serving-cert\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:41.961345 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.961427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.961391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:41.987528 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.987500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:19:41.987528 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.987520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:19:41.987665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.987500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:19:41.990069 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.990048 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:19:41.990249 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.990232 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cg2dj\"" Jul 09 16:19:41.990357 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.990341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7g2m\"" Jul 09 16:19:41.990413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:41.990389 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:19:42.062199 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbcl2\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.062199 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b10949-e84e-43ef-b4f2-4c900c949320-config\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.062435 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.062435 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.062435 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.062435 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c71ed983-0744-4a12-9b54-fd0a633b24d9-serving-cert\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.062435 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.062644 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.062644 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.062644 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-config\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.062644 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9efaa331-652f-44ca-9f42-ba4a536de051-config-volume\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.062853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661c473a-c88b-430d-b27f-191bcb18a652-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.062853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efaa331-652f-44ca-9f42-ba4a536de051-tmp-dir\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.062853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43a5652-4e16-4244-87f9-e1acaff4b3fc-cert\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.062853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edad3ea9-8b2c-4199-be52-06f37aaa936b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.062853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkqm\" (UniqueName: \"kubernetes.io/projected/c71ed983-0744-4a12-9b54-fd0a633b24d9-kube-api-access-9lkqm\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.062957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wvn\" (UniqueName: \"kubernetes.io/projected/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-kube-api-access-82wvn\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-tmp\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edad3ea9-8b2c-4199-be52-06f37aaa936b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.063129 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-snapshots\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a3a571e-9216-47f1-8c0a-22afb3279d31-serving-cert\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b10949-e84e-43ef-b4f2-4c900c949320-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjs9\" (UniqueName: \"kubernetes.io/projected/de8718da-a2ef-49b0-82ca-7f31e46519b4-kube-api-access-jsjs9\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc2q\" (UniqueName: \"kubernetes.io/projected/edad3ea9-8b2c-4199-be52-06f37aaa936b-kube-api-access-7cc2q\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhl8\" (UniqueName: \"kubernetes.io/projected/6a3a571e-9216-47f1-8c0a-22afb3279d31-kube-api-access-wbhl8\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6np\" (UniqueName: \"kubernetes.io/projected/661c473a-c88b-430d-b27f-191bcb18a652-kube-api-access-wd6np\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-stats-auth\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/fce00d72-29d4-4fc0-8ddd-8b20f4a39a39-kube-api-access-fk88c\") pod \"volume-data-source-validator-78495597f6-qdqml\" (UID: \"fce00d72-29d4-4fc0-8ddd-8b20f4a39a39\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.064572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgv5\" (UniqueName: \"kubernetes.io/projected/b8b10949-e84e-43ef-b4f2-4c900c949320-kube-api-access-bmgv5\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-trusted-ca\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744h4\" (UniqueName: \"kubernetes.io/projected/8f94b785-de6c-4cc6-8718-9925cb0619db-kube-api-access-744h4\") pod \"network-check-source-869dcfcdcd-8rxpf\" (UID: \"8f94b785-de6c-4cc6-8718-9925cb0619db\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7gl\" (UniqueName: \"kubernetes.io/projected/9efaa331-652f-44ca-9f42-ba4a536de051-kube-api-access-4c7gl\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghtkt\" (UniqueName: \"kubernetes.io/projected/d43a5652-4e16-4244-87f9-e1acaff4b3fc-kube-api-access-ghtkt\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-default-certificate\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.063805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-snapshots\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.064072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71ed983-0744-4a12-9b54-fd0a633b24d9-tmp\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.064181 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.064197 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.064276 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.564259346 +0000 UTC m=+37.088583656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.064571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.065269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.065034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71ed983-0744-4a12-9b54-fd0a633b24d9-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.067201 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.067180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c71ed983-0744-4a12-9b54-fd0a633b24d9-serving-cert\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.067269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.067253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.067303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.067276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.072731 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.072563 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbcl2\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.072822 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.072656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.072822 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.072687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkqm\" (UniqueName: \"kubernetes.io/projected/c71ed983-0744-4a12-9b54-fd0a633b24d9-kube-api-access-9lkqm\") pod \"insights-operator-68d9dd94cf-kgtns\" (UID: \"c71ed983-0744-4a12-9b54-fd0a633b24d9\") " pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.165052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.164910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wvn\" (UniqueName: \"kubernetes.io/projected/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-kube-api-access-82wvn\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.165052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.164980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.165052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edad3ea9-8b2c-4199-be52-06f37aaa936b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.165052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a3a571e-9216-47f1-8c0a-22afb3279d31-serving-cert\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b10949-e84e-43ef-b4f2-4c900c949320-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjs9\" (UniqueName: \"kubernetes.io/projected/de8718da-a2ef-49b0-82ca-7f31e46519b4-kube-api-access-jsjs9\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc2q\" (UniqueName: \"kubernetes.io/projected/edad3ea9-8b2c-4199-be52-06f37aaa936b-kube-api-access-7cc2q\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhl8\" (UniqueName: \"kubernetes.io/projected/6a3a571e-9216-47f1-8c0a-22afb3279d31-kube-api-access-wbhl8\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.165153 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.165215 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.165233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.665214683 +0000 UTC m=+37.189538994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.165264 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.665247149 +0000 UTC m=+37.189571459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6np\" (UniqueName: \"kubernetes.io/projected/661c473a-c88b-430d-b27f-191bcb18a652-kube-api-access-wd6np\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-stats-auth\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.165380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/fce00d72-29d4-4fc0-8ddd-8b20f4a39a39-kube-api-access-fk88c\") pod \"volume-data-source-validator-78495597f6-qdqml\" (UID: \"fce00d72-29d4-4fc0-8ddd-8b20f4a39a39\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgv5\" (UniqueName: \"kubernetes.io/projected/b8b10949-e84e-43ef-b4f2-4c900c949320-kube-api-access-bmgv5\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-trusted-ca\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744h4\" (UniqueName: \"kubernetes.io/projected/8f94b785-de6c-4cc6-8718-9925cb0619db-kube-api-access-744h4\") pod \"network-check-source-869dcfcdcd-8rxpf\" (UID: \"8f94b785-de6c-4cc6-8718-9925cb0619db\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7gl\" (UniqueName: \"kubernetes.io/projected/9efaa331-652f-44ca-9f42-ba4a536de051-kube-api-access-4c7gl\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghtkt\" (UniqueName: \"kubernetes.io/projected/d43a5652-4e16-4244-87f9-e1acaff4b3fc-kube-api-access-ghtkt\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-default-certificate\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b10949-e84e-43ef-b4f2-4c900c949320-config\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edad3ea9-8b2c-4199-be52-06f37aaa936b-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-config\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9efaa331-652f-44ca-9f42-ba4a536de051-config-volume\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661c473a-c88b-430d-b27f-191bcb18a652-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efaa331-652f-44ca-9f42-ba4a536de051-tmp-dir\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.166162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.165907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43a5652-4e16-4244-87f9-e1acaff4b3fc-cert\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.166058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edad3ea9-8b2c-4199-be52-06f37aaa936b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.166423 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.166482 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.666467328 +0000 UTC m=+37.190791645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.166734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-trusted-ca\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.166851 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:42.166890 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.166890 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.66687771 +0000 UTC m=+37.191202020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:42.167230 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.166995 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:42.666983758 +0000 UTC m=+37.191308093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:42.167230 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.167135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efaa331-652f-44ca-9f42-ba4a536de051-tmp-dir\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.167332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.167231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/661c473a-c88b-430d-b27f-191bcb18a652-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.167776 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.167745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8b10949-e84e-43ef-b4f2-4c900c949320-config\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.167888 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.167842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a3a571e-9216-47f1-8c0a-22afb3279d31-config\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.168024 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.167996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9efaa331-652f-44ca-9f42-ba4a536de051-config-volume\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.168571 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.168546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a3a571e-9216-47f1-8c0a-22afb3279d31-serving-cert\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.168571 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.168549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-stats-auth\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.168571 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.168587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-default-certificate\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.168821 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.168636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edad3ea9-8b2c-4199-be52-06f37aaa936b-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.169145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.169124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b10949-e84e-43ef-b4f2-4c900c949320-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.170040 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.170022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43a5652-4e16-4244-87f9-e1acaff4b3fc-cert\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.176185 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.176122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgv5\" (UniqueName: \"kubernetes.io/projected/b8b10949-e84e-43ef-b4f2-4c900c949320-kube-api-access-bmgv5\") pod \"service-ca-operator-7695fc6c8c-cdlzl\" (UID: \"b8b10949-e84e-43ef-b4f2-4c900c949320\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.176340 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.176302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghtkt\" (UniqueName: \"kubernetes.io/projected/d43a5652-4e16-4244-87f9-e1acaff4b3fc-kube-api-access-ghtkt\") pod \"ingress-canary-qnmkm\" (UID: \"d43a5652-4e16-4244-87f9-e1acaff4b3fc\") " pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.176535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.176511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6np\" (UniqueName: \"kubernetes.io/projected/661c473a-c88b-430d-b27f-191bcb18a652-kube-api-access-wd6np\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.176896 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.176871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc2q\" (UniqueName: \"kubernetes.io/projected/edad3ea9-8b2c-4199-be52-06f37aaa936b-kube-api-access-7cc2q\") pod \"kube-storage-version-migrator-operator-799cb7f958-gp6cl\" (UID: \"edad3ea9-8b2c-4199-be52-06f37aaa936b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.177145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.177123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk88c\" (UniqueName: \"kubernetes.io/projected/fce00d72-29d4-4fc0-8ddd-8b20f4a39a39-kube-api-access-fk88c\") pod \"volume-data-source-validator-78495597f6-qdqml\" (UID: \"fce00d72-29d4-4fc0-8ddd-8b20f4a39a39\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" Jul 09 16:19:42.178000 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.177958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhl8\" (UniqueName: \"kubernetes.io/projected/6a3a571e-9216-47f1-8c0a-22afb3279d31-kube-api-access-wbhl8\") pod \"console-operator-7866497494-6b2fz\" (UID: \"6a3a571e-9216-47f1-8c0a-22afb3279d31\") " pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.178386 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.178366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744h4\" (UniqueName: \"kubernetes.io/projected/8f94b785-de6c-4cc6-8718-9925cb0619db-kube-api-access-744h4\") pod \"network-check-source-869dcfcdcd-8rxpf\" (UID: \"8f94b785-de6c-4cc6-8718-9925cb0619db\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" Jul 09 16:19:42.178552 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.178535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7gl\" (UniqueName: \"kubernetes.io/projected/9efaa331-652f-44ca-9f42-ba4a536de051-kube-api-access-4c7gl\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.178621 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.178604 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjs9\" (UniqueName: \"kubernetes.io/projected/de8718da-a2ef-49b0-82ca-7f31e46519b4-kube-api-access-jsjs9\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.179019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.179002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wvn\" (UniqueName: \"kubernetes.io/projected/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-kube-api-access-82wvn\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.194353 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.194334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" Jul 09 16:19:42.217539 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.217509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" Jul 09 16:19:42.243026 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.242990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" Jul 09 16:19:42.248491 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.248463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" Jul 09 16:19:42.258404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.257235 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="48a67fc6f8618b6caaf34f5db14699eb8ffd3c3f96fabe54323eebc8e95900c0" exitCode=0 Jul 09 16:19:42.258404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.257289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"48a67fc6f8618b6caaf34f5db14699eb8ffd3c3f96fabe54323eebc8e95900c0"} Jul 09 16:19:42.266136 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.265824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:19:42.268380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.268287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" Jul 09 16:19:42.274909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.274534 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qnmkm" Jul 09 16:19:42.393312 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.388877 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-kgtns"] Jul 09 16:19:42.398411 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.398347 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71ed983_0744_4a12_9b54_fd0a633b24d9.slice/crio-11162d5d03a0c64e663cbc3aac53950c37fd830c357bea2e9db25234ddad8c6d WatchSource:0}: Error finding container 11162d5d03a0c64e663cbc3aac53950c37fd830c357bea2e9db25234ddad8c6d: Status 404 returned error can't find the container with id 11162d5d03a0c64e663cbc3aac53950c37fd830c357bea2e9db25234ddad8c6d Jul 09 16:19:42.420506 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.420426 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml"] Jul 09 16:19:42.427802 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.427768 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce00d72_29d4_4fc0_8ddd_8b20f4a39a39.slice/crio-7796c29cf5aaf8d5e0730e5b12b0c1fc402ad0c75fa13131011a976afea195a6 WatchSource:0}: Error finding container 7796c29cf5aaf8d5e0730e5b12b0c1fc402ad0c75fa13131011a976afea195a6: Status 404 returned error can't find the container with id 7796c29cf5aaf8d5e0730e5b12b0c1fc402ad0c75fa13131011a976afea195a6 Jul 09 16:19:42.481895 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.477957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl"] Jul 09 16:19:42.491902 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.491855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl"] Jul 09 16:19:42.499344 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.499316 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b10949_e84e_43ef_b4f2_4c900c949320.slice/crio-48e9f6409da6acfb3671440ad593e2d7c4603e8ba4ec257e3427af68be2b651f WatchSource:0}: Error finding container 48e9f6409da6acfb3671440ad593e2d7c4603e8ba4ec257e3427af68be2b651f: Status 404 returned error can't find the container with id 48e9f6409da6acfb3671440ad593e2d7c4603e8ba4ec257e3427af68be2b651f Jul 09 16:19:42.524519 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.524490 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qnmkm"] Jul 09 16:19:42.527830 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.527802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-6b2fz"] Jul 09 16:19:42.527969 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.527873 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43a5652_4e16_4244_87f9_e1acaff4b3fc.slice/crio-4ef557c0a13d178529c5acffbb0a22f26a208057fdc0dd228993f86ed1fcd0b4 WatchSource:0}: Error finding container 4ef557c0a13d178529c5acffbb0a22f26a208057fdc0dd228993f86ed1fcd0b4: Status 404 returned error can't find the container with id 4ef557c0a13d178529c5acffbb0a22f26a208057fdc0dd228993f86ed1fcd0b4 Jul 09 16:19:42.531062 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.531032 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3a571e_9216_47f1_8c0a_22afb3279d31.slice/crio-eba33c161cf67122d380d5ec1b79fa9fdb5ff9223e2520bbf73cb81e69508a27 WatchSource:0}: Error finding container eba33c161cf67122d380d5ec1b79fa9fdb5ff9223e2520bbf73cb81e69508a27: Status 404 returned error can't find the container with id eba33c161cf67122d380d5ec1b79fa9fdb5ff9223e2520bbf73cb81e69508a27 Jul 09 16:19:42.533962 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.533901 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf"] Jul 09 16:19:42.538848 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:42.538819 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f94b785_de6c_4cc6_8718_9925cb0619db.slice/crio-99dff5b651b14e454d9241ccd0ed4672d590780994c524fa9c7c69fc2b4ef870 WatchSource:0}: Error finding container 99dff5b651b14e454d9241ccd0ed4672d590780994c524fa9c7c69fc2b4ef870: Status 404 returned error can't find the container with id 99dff5b651b14e454d9241ccd0ed4672d590780994c524fa9c7c69fc2b4ef870 Jul 09 16:19:42.579035 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.578997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:42.579202 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.579173 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:42.579202 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.579195 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:42.579318 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.579264 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.579243494 +0000 UTC m=+38.103567818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:42.679861 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.679772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.679861 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.679810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:42.679861 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.679834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.679951 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.679967 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680005 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.679985241 +0000 UTC m=+38.204309553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680029 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.680020942 +0000 UTC m=+38.204345251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680043 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.680035618 +0000 UTC m=+38.204359932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.680066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:42.680119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:42.680165 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680160 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:42.680437 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680187 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.680176414 +0000 UTC m=+38.204500724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:42.680437 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680228 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:42.680437 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:42.680268 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:43.68025315 +0000 UTC m=+38.204577461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:43.261963 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.261904 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" event={"ID":"fce00d72-29d4-4fc0-8ddd-8b20f4a39a39","Type":"ContainerStarted","Data":"7796c29cf5aaf8d5e0730e5b12b0c1fc402ad0c75fa13131011a976afea195a6"} Jul 09 16:19:43.263649 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.263580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" event={"ID":"c71ed983-0744-4a12-9b54-fd0a633b24d9","Type":"ContainerStarted","Data":"11162d5d03a0c64e663cbc3aac53950c37fd830c357bea2e9db25234ddad8c6d"} Jul 09 16:19:43.265371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.265283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" event={"ID":"b8b10949-e84e-43ef-b4f2-4c900c949320","Type":"ContainerStarted","Data":"48e9f6409da6acfb3671440ad593e2d7c4603e8ba4ec257e3427af68be2b651f"} Jul 09 16:19:43.267565 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.267535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnmkm" event={"ID":"d43a5652-4e16-4244-87f9-e1acaff4b3fc","Type":"ContainerStarted","Data":"4ef557c0a13d178529c5acffbb0a22f26a208057fdc0dd228993f86ed1fcd0b4"} Jul 09 16:19:43.268974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.268944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" event={"ID":"8f94b785-de6c-4cc6-8718-9925cb0619db","Type":"ContainerStarted","Data":"99dff5b651b14e454d9241ccd0ed4672d590780994c524fa9c7c69fc2b4ef870"} Jul 09 16:19:43.270322 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.270236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6b2fz" event={"ID":"6a3a571e-9216-47f1-8c0a-22afb3279d31","Type":"ContainerStarted","Data":"eba33c161cf67122d380d5ec1b79fa9fdb5ff9223e2520bbf73cb81e69508a27"} Jul 09 16:19:43.272108 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.272060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" event={"ID":"edad3ea9-8b2c-4199-be52-06f37aaa936b","Type":"ContainerStarted","Data":"edecd1ad7939bc2a92b9a6fd88b0cb65315a9bf96b248be47c6a9a683a7d53ed"} Jul 09 16:19:43.587665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.587561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:43.587870 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.587850 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:43.587980 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.587873 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:43.587980 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.587947 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.587912032 +0000 UTC m=+40.112236362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:43.688882 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.688844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:43.689070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.688896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:43.689070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.688950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:43.689070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.689025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:43.689230 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:43.689084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:43.689230 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689178 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:43.689230 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689222 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:43.689373 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689256 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.689233713 +0000 UTC m=+40.213558028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:43.689373 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689276 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.689266444 +0000 UTC m=+40.213590758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:43.689373 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.689305494 +0000 UTC m=+40.213629811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:43.689531 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689379 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:43.689531 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689408 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.689397855 +0000 UTC m=+40.213722172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:43.689531 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689461 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:43.689531 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:43.689491 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:45.689480252 +0000 UTC m=+40.213804568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:45.281872 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.281826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" event={"ID":"fce00d72-29d4-4fc0-8ddd-8b20f4a39a39","Type":"ContainerStarted","Data":"2d8654eaf05318784235606c2a8339e79b1eef6af626e2ee723a4b2f2651ae93"} Jul 09 16:19:45.300444 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.300187 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-qdqml" podStartSLOduration=13.282587433 podStartE2EDuration="15.300164642s" totalImagesPullingTime="2.017577209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:45.298541231 +0000 UTC m=+39.822865555" watchObservedRunningTime="2026-07-09 16:19:45.300164642 +0000 UTC m=+39.824488972" Jul 09 16:19:45.396750 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.396685 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:45.490690 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.490654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:45.608648 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.608563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:45.608814 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.608719 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:45.608814 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.608737 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:45.608814 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.608799 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.608779298 +0000 UTC m=+44.133103612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:45.709814 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.709774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:45.710030 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.709879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:45.710030 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.709908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:45.710030 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.709958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:45.710030 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:45.710024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:45.710220 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710177 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:45.710270 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710245 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.710225076 +0000 UTC m=+44.234549386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:45.710535 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710443 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:45.710535 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710459 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:45.710535 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710504 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.710488116 +0000 UTC m=+44.234812440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:45.710535 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710532 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.710512974 +0000 UTC m=+44.234837289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:45.710535 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710463 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:45.710796 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710578 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.71056041 +0000 UTC m=+44.234884722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:45.710796 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:45.710608 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:49.71059418 +0000 UTC m=+44.234918496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:47.978358 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:47.978321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:48.046026 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:48.045985 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:49.648122 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.648081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:49.648539 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.648247 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:49.648539 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.648270 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:49.648539 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.648327 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.648312064 +0000 UTC m=+52.172636374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:49.718900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.718863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:49.749210 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.749169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:49.749383 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.749219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:49.749383 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749315 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:49.749383 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749316 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:49.749383 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749374 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.749360303 +0000 UTC m=+52.273684613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749387 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.749381564 +0000 UTC m=+52.273705874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.749422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.749443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.749466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749534 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749551 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749567 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.749555336 +0000 UTC m=+52.273879646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:49.749575 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749580 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.749572705 +0000 UTC m=+52.273897015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:49.749978 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:49.749591 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.74958548 +0000 UTC m=+52.273909790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:49.821966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:49.821910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:51.632649 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:51.632615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:51.718328 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:51.718281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:51.795720 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:51.795681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:51.872868 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:51.872837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:51.974816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:51.974714 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:52.667542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:52.667496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:52.765235 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:52.765204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.070151 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.070117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.134360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.134285 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.317371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.317337 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="d5945a81eaa466999cfcfb3bfd4ddc3f2ebf25d50d69f3f26f9514e8a867803e" exitCode=0 Jul 09 16:19:53.317561 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.317429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"d5945a81eaa466999cfcfb3bfd4ddc3f2ebf25d50d69f3f26f9514e8a867803e"} Jul 09 16:19:53.320471 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.319443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" event={"ID":"edad3ea9-8b2c-4199-be52-06f37aaa936b","Type":"ContainerStarted","Data":"f4b48d3d78130f18544ae349af0176675a151c0fee23e13eeddbe4a0d63f073a"} Jul 09 16:19:53.321427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.321399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" event={"ID":"c71ed983-0744-4a12-9b54-fd0a633b24d9","Type":"ContainerStarted","Data":"77df450f40b2559f7622811d1cba8fa2979c2d2283da8d8fc912ce8c0b41fdaa"} Jul 09 16:19:53.323368 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.323344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" event={"ID":"b8b10949-e84e-43ef-b4f2-4c900c949320","Type":"ContainerStarted","Data":"61705161fbb16fe2c15b9caeffd6fbbd5cad36eed9d37927767c91596138fa0d"} Jul 09 16:19:53.325080 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.325056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qnmkm" event={"ID":"d43a5652-4e16-4244-87f9-e1acaff4b3fc","Type":"ContainerStarted","Data":"0fdffd803e203c3cf4bd278d593000c66ca2abd77fc28a5080f5077750db03c4"} Jul 09 16:19:53.326685 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.326662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" event={"ID":"8f94b785-de6c-4cc6-8718-9925cb0619db","Type":"ContainerStarted","Data":"fde6665cd86c86f30edbec77aa08b2c1b46a964b4ae27ad1296c9a36cd54fcbf"} Jul 09 16:19:53.328390 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.328372 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/0.log" Jul 09 16:19:53.328492 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.328410 2577 generic.go:356] "Generic (PLEG): container finished" podID="6a3a571e-9216-47f1-8c0a-22afb3279d31" containerID="dd90334561a9fdf231578427ed9d0d504b18cc5672bbe2730aa5ebafbeeffdb9" exitCode=255 Jul 09 16:19:53.328492 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.328437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6b2fz" event={"ID":"6a3a571e-9216-47f1-8c0a-22afb3279d31","Type":"ContainerDied","Data":"dd90334561a9fdf231578427ed9d0d504b18cc5672bbe2730aa5ebafbeeffdb9"} Jul 09 16:19:53.328709 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.328693 2577 scope.go:117] "RemoveContainer" containerID="dd90334561a9fdf231578427ed9d0d504b18cc5672bbe2730aa5ebafbeeffdb9" Jul 09 16:19:53.352130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.352093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.376575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.374100 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" podStartSLOduration=13.583331449 podStartE2EDuration="23.374080009s" totalImagesPullingTime="9.79074856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:53.373061046 +0000 UTC m=+47.897385377" watchObservedRunningTime="2026-07-09 16:19:53.374080009 +0000 UTC m=+47.898404340" Jul 09 16:19:53.376575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.375416 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv"] Jul 09 16:19:53.400855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.400554 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" podStartSLOduration=13.731373675 podStartE2EDuration="23.400537307s" totalImagesPullingTime="9.669163632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:53.399438704 +0000 UTC m=+47.923763042" watchObservedRunningTime="2026-07-09 16:19:53.400537307 +0000 UTC m=+47.924861639" Jul 09 16:19:53.408253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.407684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" Jul 09 16:19:53.408253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.407581 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv"] Jul 09 16:19:53.412274 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.412250 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 16:19:53.413619 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.413598 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n4ft4\"" Jul 09 16:19:53.414815 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.414221 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 16:19:53.441157 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.441034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.470158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.468891 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8rxpf" podStartSLOduration=13.807031316 podStartE2EDuration="23.46887497s" totalImagesPullingTime="9.661843654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:53.467996448 +0000 UTC m=+47.992320793" watchObservedRunningTime="2026-07-09 16:19:53.46887497 +0000 UTC m=+47.993199280" Jul 09 16:19:53.470158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.469017 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qnmkm" podStartSLOduration=2.809922088 podStartE2EDuration="12.469009161s" totalImagesPullingTime="9.659087073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:53.448081758 +0000 UTC m=+47.972406091" watchObservedRunningTime="2026-07-09 16:19:53.469009161 +0000 UTC m=+47.993333494" Jul 09 16:19:53.495293 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.495183 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" podStartSLOduration=13.804971261 podStartE2EDuration="23.495160328s" totalImagesPullingTime="9.690189067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:53.495141438 +0000 UTC m=+48.019465772" watchObservedRunningTime="2026-07-09 16:19:53.495160328 +0000 UTC m=+48.019484662" Jul 09 16:19:53.525760 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.525722 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:53.590228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.590143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8rf\" (UniqueName: \"kubernetes.io/projected/03da5664-7df4-415e-bd93-498c1fb2734e-kube-api-access-nv8rf\") pod \"migrator-7f59b54f6c-hh2gv\" (UID: \"03da5664-7df4-415e-bd93-498c1fb2734e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" Jul 09 16:19:53.690976 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.690901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8rf\" (UniqueName: \"kubernetes.io/projected/03da5664-7df4-415e-bd93-498c1fb2734e-kube-api-access-nv8rf\") pod \"migrator-7f59b54f6c-hh2gv\" (UID: \"03da5664-7df4-415e-bd93-498c1fb2734e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" Jul 09 16:19:53.701840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.701811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8rf\" (UniqueName: \"kubernetes.io/projected/03da5664-7df4-415e-bd93-498c1fb2734e-kube-api-access-nv8rf\") pod \"migrator-7f59b54f6c-hh2gv\" (UID: \"03da5664-7df4-415e-bd93-498c1fb2734e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" Jul 09 16:19:53.731877 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.731846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" Jul 09 16:19:53.895441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:53.895333 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv"] Jul 09 16:19:53.898085 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:53.898059 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03da5664_7df4_415e_bd93_498c1fb2734e.slice/crio-d8b549c637df7e256202884f3f16f36963e7230b47332a6d9ac85158b8b5e50d WatchSource:0}: Error finding container d8b549c637df7e256202884f3f16f36963e7230b47332a6d9ac85158b8b5e50d: Status 404 returned error can't find the container with id d8b549c637df7e256202884f3f16f36963e7230b47332a6d9ac85158b8b5e50d Jul 09 16:19:54.220647 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.220570 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9dffg"] Jul 09 16:19:54.254538 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.254504 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9dffg"] Jul 09 16:19:54.254682 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.254647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.257706 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.257682 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 16:19:54.257847 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.257682 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 16:19:54.257847 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.257755 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pd94\"" Jul 09 16:19:54.294751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.294714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.294904 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.294760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2gr\" (UniqueName: \"kubernetes.io/projected/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-api-access-8h2gr\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.294904 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.294831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-data-volume\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.294904 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.294884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-crio-socket\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.295052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.295027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.333208 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.333171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" event={"ID":"03da5664-7df4-415e-bd93-498c1fb2734e","Type":"ContainerStarted","Data":"d8b549c637df7e256202884f3f16f36963e7230b47332a6d9ac85158b8b5e50d"} Jul 09 16:19:54.334699 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.334679 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:19:54.335106 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.335090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/0.log" Jul 09 16:19:54.335163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.335125 2577 generic.go:356] "Generic (PLEG): container finished" podID="6a3a571e-9216-47f1-8c0a-22afb3279d31" containerID="edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079" exitCode=255 Jul 09 16:19:54.335210 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.335196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6b2fz" event={"ID":"6a3a571e-9216-47f1-8c0a-22afb3279d31","Type":"ContainerDied","Data":"edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079"} Jul 09 16:19:54.338231 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.338207 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88" containerID="ce9e51d6e4f774441e130e36b0d159735d06ee6feb836b9c2d53acc25bc5c3fa" exitCode=0 Jul 09 16:19:54.338330 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.338308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerDied","Data":"ce9e51d6e4f774441e130e36b0d159735d06ee6feb836b9c2d53acc25bc5c3fa"} Jul 09 16:19:54.345981 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.345962 2577 scope.go:117] "RemoveContainer" containerID="dd90334561a9fdf231578427ed9d0d504b18cc5672bbe2730aa5ebafbeeffdb9" Jul 09 16:19:54.346079 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.346063 2577 scope.go:117] "RemoveContainer" containerID="edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079" Jul 09 16:19:54.346239 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.346219 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6b2fz_openshift-console-operator(6a3a571e-9216-47f1-8c0a-22afb3279d31)\"" pod="openshift-console-operator/console-operator-7866497494-6b2fz" podUID="6a3a571e-9216-47f1-8c0a-22afb3279d31" Jul 09 16:19:54.396729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.396525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.396729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.396590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2gr\" (UniqueName: \"kubernetes.io/projected/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-api-access-8h2gr\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.396729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.396626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-data-volume\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.396729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.396669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-crio-socket\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.397162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.397137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-crio-socket\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.397238 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.397197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-data-volume\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.397321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.397298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.397656 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.397635 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:19:54.397819 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.397799 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:54.897776944 +0000 UTC m=+49.422101272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : secret "insights-runtime-extractor-tls" not found Jul 09 16:19:54.397937 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.397896 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:19:54.398014 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.397994 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:54.897973721 +0000 UTC m=+49.422298034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : configmap "kube-rbac-proxy" not found Jul 09 16:19:54.405475 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.405446 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2gr\" (UniqueName: \"kubernetes.io/projected/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-api-access-8h2gr\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.621874 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.621839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:54.721840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.721759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:54.902895 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.902842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.903152 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.903021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:54.903152 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.903036 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:19:54.903152 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.903124 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:19:54.903152 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.903141 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:55.903119602 +0000 UTC m=+50.427443913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : secret "insights-runtime-extractor-tls" not found Jul 09 16:19:54.903409 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:54.903181 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:55.903162048 +0000 UTC m=+50.427486359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : configmap "kube-rbac-proxy" not found Jul 09 16:19:54.975786 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.975751 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7nvdj"] Jul 09 16:19:54.989684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.989652 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:54.992505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.992469 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 16:19:54.992505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.992486 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 16:19:54.992679 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.992491 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-64d84\"" Jul 09 16:19:54.999783 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:54.999714 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7nvdj"] Jul 09 16:19:55.004059 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.004035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.004176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.004151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.105540 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.105503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.105722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.105659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.105857 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.105843 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 16:19:55.105940 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.105902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert podName:9c3a8f6e-1b40-4d45-894f-3d7e6bed442d nodeName:}" failed. No retries permitted until 2026-07-09 16:19:55.605883851 +0000 UTC m=+50.130208164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-7nvdj" (UID: "9c3a8f6e-1b40-4d45-894f-3d7e6bed442d") : secret "networking-console-plugin-cert" not found Jul 09 16:19:55.106676 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.106235 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.169055 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.168968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:55.232105 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.232069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:55.343017 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.342983 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:19:55.343399 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.343368 2577 scope.go:117] "RemoveContainer" containerID="edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079" Jul 09 16:19:55.343590 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.343565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6b2fz_openshift-console-operator(6a3a571e-9216-47f1-8c0a-22afb3279d31)\"" pod="openshift-console-operator/console-operator-7866497494-6b2fz" podUID="6a3a571e-9216-47f1-8c0a-22afb3279d31" Jul 09 16:19:55.347130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.347103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" event={"ID":"7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88","Type":"ContainerStarted","Data":"c874fba5d4d4345482b27bd6d7840a117f90b6eeb9aac2a52ed69e3d5c8eb1c8"} Jul 09 16:19:55.382288 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.382235 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j4j5t" podStartSLOduration=8.438873131 podStartE2EDuration="49.382217512s" totalImagesPullingTime="40.943344381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:55.381589138 +0000 UTC m=+49.905913495" watchObservedRunningTime="2026-07-09 16:19:55.382217512 +0000 UTC m=+49.906541844" Jul 09 16:19:55.610905 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.610863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:55.611166 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.611034 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 16:19:55.611166 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.611100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert podName:9c3a8f6e-1b40-4d45-894f-3d7e6bed442d nodeName:}" failed. No retries permitted until 2026-07-09 16:19:56.611084199 +0000 UTC m=+51.135408511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-7nvdj" (UID: "9c3a8f6e-1b40-4d45-894f-3d7e6bed442d") : secret "networking-console-plugin-cert" not found Jul 09 16:19:55.913461 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.913426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:55.913761 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.913565 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:19:55.913761 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.913620 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.913604337 +0000 UTC m=+52.437928647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : secret "insights-runtime-extractor-tls" not found Jul 09 16:19:55.913761 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.913629 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:19:55.913761 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:55.913688 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:19:57.913674317 +0000 UTC m=+52.437998628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : configmap "kube-rbac-proxy" not found Jul 09 16:19:55.913761 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.913568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:55.940830 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.940806 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bvw2v"] Jul 09 16:19:55.971322 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.971290 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bvw2v"] Jul 09 16:19:55.971454 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.971384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:55.976236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.976206 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 16:19:55.976378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.976245 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 16:19:55.976378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.976270 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 16:19:55.976378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.976301 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pcmfh\"" Jul 09 16:19:55.980198 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.980179 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 16:19:55.995681 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:55.995653 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:56.014946 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.014899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6wf\" (UniqueName: \"kubernetes.io/projected/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-kube-api-access-vk6wf\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.015095 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.015064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-key\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.015180 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.015116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-cabundle\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.066701 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.066669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:56.116118 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.116078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-key\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.116326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.116144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-cabundle\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.116326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.116262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6wf\" (UniqueName: \"kubernetes.io/projected/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-kube-api-access-vk6wf\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.116748 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.116720 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8snr_df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87/dns-node-resolver/0.log" Jul 09 16:19:56.116938 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.116899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-cabundle\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.118773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.118747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-signing-key\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.125169 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.125111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6wf\" (UniqueName: \"kubernetes.io/projected/8a0a204d-4bb9-4e92-8c86-e27fc7e27777-kube-api-access-vk6wf\") pod \"service-ca-79f68b9f4f-bvw2v\" (UID: \"8a0a204d-4bb9-4e92-8c86-e27fc7e27777\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.167769 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.167732 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:56.288106 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.288073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" Jul 09 16:19:56.353222 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.353190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" event={"ID":"03da5664-7df4-415e-bd93-498c1fb2734e","Type":"ContainerStarted","Data":"5a3f8841fbc8b791c80c96a304782c99c2446de276c3194ac4c955504db99b80"} Jul 09 16:19:56.353388 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.353245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" event={"ID":"03da5664-7df4-415e-bd93-498c1fb2734e","Type":"ContainerStarted","Data":"64f09867aa65fa2afc1f35ebc482a3833c448603c27b0950e459bcfdff12456f"} Jul 09 16:19:56.371266 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.371218 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-hh2gv" podStartSLOduration=1.511552209 podStartE2EDuration="3.371199099s" totalImagesPullingTime="1.85964689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:56.369896764 +0000 UTC m=+50.894221095" watchObservedRunningTime="2026-07-09 16:19:56.371199099 +0000 UTC m=+50.895523457" Jul 09 16:19:56.438398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.438374 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bvw2v"] Jul 09 16:19:56.440672 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:19:56.440642 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0a204d_4bb9_4e92_8c86_e27fc7e27777.slice/crio-40a852774eebc8fe229551f87e8aee1214e12a0747f9d351ad0e5e3e7fa84a22 WatchSource:0}: Error finding container 40a852774eebc8fe229551f87e8aee1214e12a0747f9d351ad0e5e3e7fa84a22: Status 404 returned error can't find the container with id 40a852774eebc8fe229551f87e8aee1214e12a0747f9d351ad0e5e3e7fa84a22 Jul 09 16:19:56.463586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.463552 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:56.546390 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.546357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:56.621889 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.621851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:56.622108 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:56.622053 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 16:19:56.622175 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:56.622131 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert podName:9c3a8f6e-1b40-4d45-894f-3d7e6bed442d nodeName:}" failed. No retries permitted until 2026-07-09 16:19:58.622110978 +0000 UTC m=+53.146435299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-7nvdj" (UID: "9c3a8f6e-1b40-4d45-894f-3d7e6bed442d") : secret "networking-console-plugin-cert" not found Jul 09 16:19:56.874481 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:56.874443 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:57.358767 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.358729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" event={"ID":"8a0a204d-4bb9-4e92-8c86-e27fc7e27777","Type":"ContainerStarted","Data":"bada769495ee86028f14f774d53d2dc08bf04bf270ada04f3e59f030b3cf925d"} Jul 09 16:19:57.359183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.358793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" event={"ID":"8a0a204d-4bb9-4e92-8c86-e27fc7e27777","Type":"ContainerStarted","Data":"40a852774eebc8fe229551f87e8aee1214e12a0747f9d351ad0e5e3e7fa84a22"} Jul 09 16:19:57.378442 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.378394 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-bvw2v" podStartSLOduration=2.378377459 podStartE2EDuration="2.378377459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:19:57.37771012 +0000 UTC m=+51.902034453" watchObservedRunningTime="2026-07-09 16:19:57.378377459 +0000 UTC m=+51.902701791" Jul 09 16:19:57.516116 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.516086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxlmb_3d10c6d0-5fa2-469f-87f1-913829110c26/node-ca/0.log" Jul 09 16:19:57.720064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.719975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:57.734257 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.734232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:19:57.734473 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.734355 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:19:57.734546 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.734476 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78858457bc-l66hw: secret "image-registry-tls" not found Jul 09 16:19:57.734546 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.734537 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls podName:7cfe4a52-2ba4-4d2c-b50f-100866fd3820 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.734518724 +0000 UTC m=+68.258843036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls") pod "image-registry-78858457bc-l66hw" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820") : secret "image-registry-tls" not found Jul 09 16:19:57.835600 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.835560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.835626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.835721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835735 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.835757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.835793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:19:57.835810 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835809 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls podName:de8718da-a2ef-49b0-82ca-7f31e46519b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.835788825 +0000 UTC m=+68.360113140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls") pod "cluster-samples-operator-65db858894-28vsp" (UID: "de8718da-a2ef-49b0-82ca-7f31e46519b4") : secret "samples-operator-tls" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835864 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835864 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835936 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls podName:661c473a-c88b-430d-b27f-191bcb18a652 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.835897057 +0000 UTC m=+68.360221370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-nlql6" (UID: "661c473a-c88b-430d-b27f-191bcb18a652") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls podName:9efaa331-652f-44ca-9f42-ba4a536de051 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.835944917 +0000 UTC m=+68.360269232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls") pod "dns-default-zfd8z" (UID: "9efaa331-652f-44ca-9f42-ba4a536de051") : secret "dns-default-metrics-tls" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835966 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.835996 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.835986242 +0000 UTC m=+68.360310555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : secret "router-metrics-certs-default" not found Jul 09 16:19:57.836129 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.836023 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle podName:5eb33ab3-d8d8-438c-87c5-d70acafa0db7 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:13.836015051 +0000 UTC m=+68.360339370 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle") pod "router-default-7f6db5c95b-l6zp4" (UID: "5eb33ab3-d8d8-438c-87c5-d70acafa0db7") : configmap references non-existent config key: service-ca.crt Jul 09 16:19:57.902162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.902127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:57.920445 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.920418 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnmkm_d43a5652-4e16-4244-87f9-e1acaff4b3fc/serve-healthcheck-canary/0.log" Jul 09 16:19:57.937067 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.937035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:57.937245 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.937147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:19:57.937245 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.937196 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:19:57.937364 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.937249 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:19:57.937364 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.937281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:01.93726053 +0000 UTC m=+56.461584843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : secret "insights-runtime-extractor-tls" not found Jul 09 16:19:57.937364 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:57.937303 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:01.937287492 +0000 UTC m=+56.461611805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : configmap "kube-rbac-proxy" not found Jul 09 16:19:57.998544 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:57.998514 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:58.101107 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.101071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:58.124456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.124422 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hh2gv_03da5664-7df4-415e-bd93-498c1fb2734e/migrator/0.log" Jul 09 16:19:58.202325 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.202288 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:58.318217 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.318134 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hh2gv_03da5664-7df4-415e-bd93-498c1fb2734e/graceful-termination/0.log" Jul 09 16:19:58.407311 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.407278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:58.518326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.518290 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-gp6cl_edad3ea9-8b2c-4199-be52-06f37aaa936b/kube-storage-version-migrator-operator/0.log" Jul 09 16:19:58.644885 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:58.644788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:19:58.645078 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:58.644984 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 16:19:58.645078 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:19:58.645067 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert podName:9c3a8f6e-1b40-4d45-894f-3d7e6bed442d nodeName:}" failed. No retries permitted until 2026-07-09 16:20:02.645047006 +0000 UTC m=+57.169371322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-7nvdj" (UID: "9c3a8f6e-1b40-4d45-894f-3d7e6bed442d") : secret "networking-console-plugin-cert" not found Jul 09 16:19:59.803561 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:59.803516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:59.887740 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:59.887695 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:19:59.999134 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:19:59.999093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:00.206321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:00.206230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:00.393549 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:00.393516 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:00.496005 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:00.495965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:01.574138 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.574096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:01.719183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.719150 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:01.977821 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.977787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:01.978040 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.977950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:01.978121 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:01.978100 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:20:01.978174 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:01.978166 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls podName:5b6b02ae-6d00-43c6-b119-2ee2e78f0887 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:09.978150522 +0000 UTC m=+64.502474832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dffg" (UID: "5b6b02ae-6d00-43c6-b119-2ee2e78f0887") : secret "insights-runtime-extractor-tls" not found Jul 09 16:20:01.978354 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.978337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:01.998693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:01.998660 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:02.216900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:02.216865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:02.267017 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:02.266911 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:20:02.267140 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:02.267018 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:20:02.267401 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:02.267388 2577 scope.go:117] "RemoveContainer" containerID="edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079" Jul 09 16:20:02.267554 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:02.267536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-6b2fz_openshift-console-operator(6a3a571e-9216-47f1-8c0a-22afb3279d31)\"" pod="openshift-console-operator/console-operator-7866497494-6b2fz" podUID="6a3a571e-9216-47f1-8c0a-22afb3279d31" Jul 09 16:20:02.684971 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:02.684850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:20:02.685425 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:02.685039 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 16:20:02.685425 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:02.685126 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert podName:9c3a8f6e-1b40-4d45-894f-3d7e6bed442d nodeName:}" failed. No retries permitted until 2026-07-09 16:20:10.685103706 +0000 UTC m=+65.209428023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-7nvdj" (UID: "9c3a8f6e-1b40-4d45-894f-3d7e6bed442d") : secret "networking-console-plugin-cert" not found Jul 09 16:20:03.261690 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:03.261654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:03.346489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:03.346453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:03.592315 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:03.592221 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:03.985651 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:03.985621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:04.395698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:04.395598 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:04.697135 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:04.697044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:04.796748 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:04.796713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:04.897828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:04.897795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:04.997808 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:04.997772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:05.098943 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:05.098898 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:05.387101 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:05.387005 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:06.005242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:06.005208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:06.410391 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:06.410311 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:06.641278 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:06.641239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:07.190304 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:07.190271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:07.257754 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:07.257727 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8482k" Jul 09 16:20:07.710094 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:07.710063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:08.004268 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:08.004238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:08.186592 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:08.186555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:08.400376 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:08.400293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:08.501321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:08.501286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:10.053960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.053899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:10.056383 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.056351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b6b02ae-6d00-43c6-b119-2ee2e78f0887-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dffg\" (UID: \"5b6b02ae-6d00-43c6-b119-2ee2e78f0887\") " pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:10.167491 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.167460 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pd94\"" Jul 09 16:20:10.174801 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.174772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9dffg" Jul 09 16:20:10.315969 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.315907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9dffg"] Jul 09 16:20:10.317909 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:10.317880 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6b02ae_6d00_43c6_b119_2ee2e78f0887.slice/crio-92aa58a4c8f3c3f81246ba74b23ce09464fe581bdbc9cddb0f6709933b7a86c2 WatchSource:0}: Error finding container 92aa58a4c8f3c3f81246ba74b23ce09464fe581bdbc9cddb0f6709933b7a86c2: Status 404 returned error can't find the container with id 92aa58a4c8f3c3f81246ba74b23ce09464fe581bdbc9cddb0f6709933b7a86c2 Jul 09 16:20:10.401318 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.401279 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dffg" event={"ID":"5b6b02ae-6d00-43c6-b119-2ee2e78f0887","Type":"ContainerStarted","Data":"92aa58a4c8f3c3f81246ba74b23ce09464fe581bdbc9cddb0f6709933b7a86c2"} Jul 09 16:20:10.761776 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.761736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:20:10.764449 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.764423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c3a8f6e-1b40-4d45-894f-3d7e6bed442d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7nvdj\" (UID: \"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:20:10.902555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.902517 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-64d84\"" Jul 09 16:20:10.910264 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:10.910234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" Jul 09 16:20:11.095496 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.095247 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7nvdj"] Jul 09 16:20:11.099350 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:11.099304 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3a8f6e_1b40_4d45_894f_3d7e6bed442d.slice/crio-96526c33e9cdff9d148a28fa28a9bb2222a41094e59409c769c099dbf2652402 WatchSource:0}: Error finding container 96526c33e9cdff9d148a28fa28a9bb2222a41094e59409c769c099dbf2652402: Status 404 returned error can't find the container with id 96526c33e9cdff9d148a28fa28a9bb2222a41094e59409c769c099dbf2652402 Jul 09 16:20:11.407466 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.407377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" event={"ID":"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d","Type":"ContainerStarted","Data":"96526c33e9cdff9d148a28fa28a9bb2222a41094e59409c769c099dbf2652402"} Jul 09 16:20:11.672789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.672701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:20:11.673003 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.672867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:20:11.675864 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.675837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:20:11.676026 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.675901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gglfg\" (UniqueName: \"kubernetes.io/projected/d739eb4d-2df5-4fd1-9bab-5ec49b834c84-kube-api-access-gglfg\") pod \"network-check-target-rd9v8\" (UID: \"d739eb4d-2df5-4fd1-9bab-5ec49b834c84\") " pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:20:11.685868 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.685836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a659e30c-c220-4d77-8159-c4d452ea8e95-metrics-certs\") pod \"network-metrics-daemon-dzjz6\" (UID: \"a659e30c-c220-4d77-8159-c4d452ea8e95\") " pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:20:11.699755 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.699728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q7g2m\"" Jul 09 16:20:11.707228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.707205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzjz6" Jul 09 16:20:11.710029 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.710011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cg2dj\"" Jul 09 16:20:11.717375 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.717350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:20:11.774499 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.774265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:20:11.777362 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.777340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:20:11.788783 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.788016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19baab78-2289-4c52-a951-febcda3d048a-original-pull-secret\") pod \"global-pull-secret-syncer-kr65g\" (UID: \"19baab78-2289-4c52-a951-febcda3d048a\") " pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:20:11.868079 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.868049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzjz6"] Jul 09 16:20:11.869977 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:11.869943 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda659e30c_c220_4d77_8159_c4d452ea8e95.slice/crio-076030867dfeb55d03ae12eb99ba31c24aaa5cc553d9dbfb9d6ec8e79d78dfb9 WatchSource:0}: Error finding container 076030867dfeb55d03ae12eb99ba31c24aaa5cc553d9dbfb9d6ec8e79d78dfb9: Status 404 returned error can't find the container with id 076030867dfeb55d03ae12eb99ba31c24aaa5cc553d9dbfb9d6ec8e79d78dfb9 Jul 09 16:20:11.887732 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:11.887640 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rd9v8"] Jul 09 16:20:12.003099 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.003064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kr65g" Jul 09 16:20:12.153972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.153939 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kr65g"] Jul 09 16:20:12.413211 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.413174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rd9v8" event={"ID":"d739eb4d-2df5-4fd1-9bab-5ec49b834c84","Type":"ContainerStarted","Data":"9505ee2b7854f75cfa538047eb9df50cab527583fcb17a3b6fd6c3c2a76c3af3"} Jul 09 16:20:12.413404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.413232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rd9v8" event={"ID":"d739eb4d-2df5-4fd1-9bab-5ec49b834c84","Type":"ContainerStarted","Data":"1ec50729847f85a43cdf78dcb6321dad6b65210d4b6ad2402041f8c0970b0413"} Jul 09 16:20:12.413404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.413282 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:20:12.414776 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:12.414744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzjz6" event={"ID":"a659e30c-c220-4d77-8159-c4d452ea8e95","Type":"ContainerStarted","Data":"076030867dfeb55d03ae12eb99ba31c24aaa5cc553d9dbfb9d6ec8e79d78dfb9"} Jul 09 16:20:13.652161 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:13.652126 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19baab78_2289_4c52_a951_febcda3d048a.slice/crio-b736797bbc9eb7f1ff12117a1015593051e05f630917467862b1ac3505cbcb6b WatchSource:0}: Error finding container b736797bbc9eb7f1ff12117a1015593051e05f630917467862b1ac3505cbcb6b: Status 404 returned error can't find the container with id b736797bbc9eb7f1ff12117a1015593051e05f630917467862b1ac3505cbcb6b Jul 09 16:20:13.793505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.793467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:20:13.796558 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.796524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"image-registry-78858457bc-l66hw\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:20:13.894296 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.894260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:13.894478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.894318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:13.894478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.894350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:20:13.894478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.894403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:20:13.894478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.894445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:20:13.895787 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.895486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-service-ca-bundle\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:13.897532 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.897477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/661c473a-c88b-430d-b27f-191bcb18a652-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-nlql6\" (UID: \"661c473a-c88b-430d-b27f-191bcb18a652\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:20:13.898441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.898412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9efaa331-652f-44ca-9f42-ba4a536de051-metrics-tls\") pod \"dns-default-zfd8z\" (UID: \"9efaa331-652f-44ca-9f42-ba4a536de051\") " pod="openshift-dns/dns-default-zfd8z" Jul 09 16:20:13.899341 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.899288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de8718da-a2ef-49b0-82ca-7f31e46519b4-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-28vsp\" (UID: \"de8718da-a2ef-49b0-82ca-7f31e46519b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:20:13.899341 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:13.899334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eb33ab3-d8d8-438c-87c5-d70acafa0db7-metrics-certs\") pod \"router-default-7f6db5c95b-l6zp4\" (UID: \"5eb33ab3-d8d8-438c-87c5-d70acafa0db7\") " pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:14.012322 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.012281 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rfjnp\"" Jul 09 16:20:14.020205 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.020171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:20:14.031127 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.031089 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5b7x6\"" Jul 09 16:20:14.038852 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.038826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" Jul 09 16:20:14.039779 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.039750 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vdvwq\"" Jul 09 16:20:14.047645 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.047619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:14.056618 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.056594 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-59jjp\"" Jul 09 16:20:14.063982 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.063950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" Jul 09 16:20:14.081174 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.081149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-466pn\"" Jul 09 16:20:14.088829 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.088801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zfd8z" Jul 09 16:20:14.424264 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.424187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kr65g" event={"ID":"19baab78-2289-4c52-a951-febcda3d048a","Type":"ContainerStarted","Data":"b736797bbc9eb7f1ff12117a1015593051e05f630917467862b1ac3505cbcb6b"} Jul 09 16:20:14.836241 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.836184 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rd9v8" podStartSLOduration=68.836166465 podStartE2EDuration="1m8.836166465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:12.429245156 +0000 UTC m=+66.953569488" watchObservedRunningTime="2026-07-09 16:20:14.836166465 +0000 UTC m=+69.360490805" Jul 09 16:20:14.836617 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.836402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:20:14.857190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.857152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6db5c95b-l6zp4"] Jul 09 16:20:14.880382 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.880348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6"] Jul 09 16:20:14.896337 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:14.896308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp"] Jul 09 16:20:14.998383 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:14.998347 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfe4a52_2ba4_4d2c_b50f_100866fd3820.slice/crio-724e67a992bc19a1d1923b1258c298984f51239d3898d57fcee7940807f343d8 WatchSource:0}: Error finding container 724e67a992bc19a1d1923b1258c298984f51239d3898d57fcee7940807f343d8: Status 404 returned error can't find the container with id 724e67a992bc19a1d1923b1258c298984f51239d3898d57fcee7940807f343d8 Jul 09 16:20:15.000049 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:15.000013 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661c473a_c88b_430d_b27f_191bcb18a652.slice/crio-a90e25d6022f4f2fbaa578665db1efebdcb4f081fb5dfee42d20f16c714807d1 WatchSource:0}: Error finding container a90e25d6022f4f2fbaa578665db1efebdcb4f081fb5dfee42d20f16c714807d1: Status 404 returned error can't find the container with id a90e25d6022f4f2fbaa578665db1efebdcb4f081fb5dfee42d20f16c714807d1 Jul 09 16:20:15.001539 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:15.001502 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb33ab3_d8d8_438c_87c5_d70acafa0db7.slice/crio-a2130f614e0e6cb5576264046bed34e60b8547b267f3a2f1568a544664b60dcd WatchSource:0}: Error finding container a2130f614e0e6cb5576264046bed34e60b8547b267f3a2f1568a544664b60dcd: Status 404 returned error can't find the container with id a2130f614e0e6cb5576264046bed34e60b8547b267f3a2f1568a544664b60dcd Jul 09 16:20:15.047707 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.047672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zfd8z"] Jul 09 16:20:15.053381 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:15.053350 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efaa331_652f_44ca_9f42_ba4a536de051.slice/crio-becf7ee6290958764be2073ea47b2c0b8812886bf4f6a20f4b12771cbcb0aadd WatchSource:0}: Error finding container becf7ee6290958764be2073ea47b2c0b8812886bf4f6a20f4b12771cbcb0aadd: Status 404 returned error can't find the container with id becf7ee6290958764be2073ea47b2c0b8812886bf4f6a20f4b12771cbcb0aadd Jul 09 16:20:15.432031 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.431884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzjz6" event={"ID":"a659e30c-c220-4d77-8159-c4d452ea8e95","Type":"ContainerStarted","Data":"9306bcc603dba484534654cc01a3ba9c87b16bab24b9b0c7af777686e3c08d08"} Jul 09 16:20:15.432031 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.431971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzjz6" event={"ID":"a659e30c-c220-4d77-8159-c4d452ea8e95","Type":"ContainerStarted","Data":"7ad09adb010c70cd0e23abb0edfa3e6a42aabfd74c47118fad14c4dc6943b098"} Jul 09 16:20:15.433485 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.433454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" event={"ID":"661c473a-c88b-430d-b27f-191bcb18a652","Type":"ContainerStarted","Data":"a90e25d6022f4f2fbaa578665db1efebdcb4f081fb5dfee42d20f16c714807d1"} Jul 09 16:20:15.434806 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.434778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" event={"ID":"de8718da-a2ef-49b0-82ca-7f31e46519b4","Type":"ContainerStarted","Data":"2fa3e0fa7fd4b0566ef4306db2b86fe675bdcc75abdd9bcad5bdf4e2fb164075"} Jul 09 16:20:15.436847 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.436788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" event={"ID":"5eb33ab3-d8d8-438c-87c5-d70acafa0db7","Type":"ContainerStarted","Data":"f805f416c4977b6d7a953a3a786c5bc645dc722841e90c386dbd1931abbb1597"} Jul 09 16:20:15.436847 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.436832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" event={"ID":"5eb33ab3-d8d8-438c-87c5-d70acafa0db7","Type":"ContainerStarted","Data":"a2130f614e0e6cb5576264046bed34e60b8547b267f3a2f1568a544664b60dcd"} Jul 09 16:20:15.438613 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.438584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78858457bc-l66hw" event={"ID":"7cfe4a52-2ba4-4d2c-b50f-100866fd3820","Type":"ContainerStarted","Data":"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632"} Jul 09 16:20:15.438716 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.438628 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78858457bc-l66hw" event={"ID":"7cfe4a52-2ba4-4d2c-b50f-100866fd3820","Type":"ContainerStarted","Data":"724e67a992bc19a1d1923b1258c298984f51239d3898d57fcee7940807f343d8"} Jul 09 16:20:15.438716 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.438706 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:20:15.440667 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.440630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dffg" event={"ID":"5b6b02ae-6d00-43c6-b119-2ee2e78f0887","Type":"ContainerStarted","Data":"25d7c9f8578cbde8a56892d2c275797ddbacf0b81eb4be16a3698dec01551edc"} Jul 09 16:20:15.442480 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.442456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" event={"ID":"9c3a8f6e-1b40-4d45-894f-3d7e6bed442d","Type":"ContainerStarted","Data":"b3b02ef689caf4028979572402c8b8f6090bd962c276a209f026bac8fb9265e5"} Jul 09 16:20:15.443735 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.443712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfd8z" event={"ID":"9efaa331-652f-44ca-9f42-ba4a536de051","Type":"ContainerStarted","Data":"becf7ee6290958764be2073ea47b2c0b8812886bf4f6a20f4b12771cbcb0aadd"} Jul 09 16:20:15.450738 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.450695 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dzjz6" podStartSLOduration=66.262395797 podStartE2EDuration="1m9.450625944s" totalImagesPullingTime="3.188230147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:15.449438423 +0000 UTC m=+69.973762757" watchObservedRunningTime="2026-07-09 16:20:15.450625944 +0000 UTC m=+69.974950276" Jul 09 16:20:15.469046 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.468997 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-7nvdj" podStartSLOduration=17.95874172 podStartE2EDuration="21.468980342s" totalImagesPullingTime="3.510238622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:15.467495497 +0000 UTC m=+69.991819830" watchObservedRunningTime="2026-07-09 16:20:15.468980342 +0000 UTC m=+69.993304673" Jul 09 16:20:15.490669 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.490318 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78858457bc-l66hw" podStartSLOduration=68.490294558 podStartE2EDuration="1m8.490294558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:15.489833249 +0000 UTC m=+70.014157582" watchObservedRunningTime="2026-07-09 16:20:15.490294558 +0000 UTC m=+70.014618889" Jul 09 16:20:15.517096 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.516990 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" podStartSLOduration=45.516969333 podStartE2EDuration="45.516969333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:15.513311354 +0000 UTC m=+70.037635688" watchObservedRunningTime="2026-07-09 16:20:15.516969333 +0000 UTC m=+70.041293666" Jul 09 16:20:15.925490 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.925422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:15.989561 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:15.989531 2577 scope.go:117] "RemoveContainer" containerID="edf60911f2a979b29db01c20028c68c580a3936a4f2ebc09ca1b8484c81d9079" Jul 09 16:20:16.027905 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.027866 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:16.048896 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.048864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:16.052569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.052514 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:16.455206 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.455115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dffg" event={"ID":"5b6b02ae-6d00-43c6-b119-2ee2e78f0887","Type":"ContainerStarted","Data":"e7213087272082afda8129c9d4183d91ace1b7920e7d501399b3de45edaf443d"} Jul 09 16:20:16.457595 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.457568 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:20:16.458286 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.458243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-6b2fz" event={"ID":"6a3a571e-9216-47f1-8c0a-22afb3279d31","Type":"ContainerStarted","Data":"89663690a56a64236476ef961a5465bd81a9b55e472cdc02b2999cbdc25e930e"} Jul 09 16:20:16.459766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.459259 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:16.459766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.459301 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:20:16.461194 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.461155 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f6db5c95b-l6zp4" Jul 09 16:20:16.500724 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.500333 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-6b2fz" podStartSLOduration=36.869171768 podStartE2EDuration="46.500308816s" totalImagesPullingTime="9.631137048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:16.479536294 +0000 UTC m=+71.003860626" watchObservedRunningTime="2026-07-09 16:20:16.500308816 +0000 UTC m=+71.024633149" Jul 09 16:20:16.775536 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.775503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-6b2fz" Jul 09 16:20:16.983537 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:16.983451 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:17.084907 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:17.084814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:17.267539 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:17.267493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:17.639648 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:17.639612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:17.723862 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:17.723821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.050523 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.050481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.157482 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.157445 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.297284 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.297246 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs"] Jul 09 16:20:18.303892 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.303861 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb"] Jul 09 16:20:18.304064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.304045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.307806 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.307778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.308399 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.308376 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 16:20:18.308515 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.308469 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 16:20:18.310088 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.310064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 16:20:18.310200 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.310091 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 16:20:18.310839 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.310820 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 16:20:18.311081 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.311062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 16:20:18.312635 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.311258 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 16:20:18.312635 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.311696 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 16:20:18.316580 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.316554 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt"] Jul 09 16:20:18.331100 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.331065 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs"] Jul 09 16:20:18.331259 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.331124 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb"] Jul 09 16:20:18.331259 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.331207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.334256 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.334229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2ngnm\"" Jul 09 16:20:18.334402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.334274 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 16:20:18.339348 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.339323 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt"] Jul 09 16:20:18.383195 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.383160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.394234 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.394206 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-h8jff"] Jul 09 16:20:18.397719 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.397691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:18.400560 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.400537 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 16:20:18.400698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.400540 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6jn8m\"" Jul 09 16:20:18.400874 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.400858 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 16:20:18.415997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.415967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-h8jff"] Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c79e93-cb2e-4279-be58-a81d91e85f1e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bf908f-26ad-4edb-9b0b-7adf4008bd00-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7czf\" (UniqueName: \"kubernetes.io/projected/15c79e93-cb2e-4279-be58-a81d91e85f1e-kube-api-access-p7czf\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbvd\" (UniqueName: \"kubernetes.io/projected/28bf908f-26ad-4edb-9b0b-7adf4008bd00-kube-api-access-rdbvd\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5562a37-298a-406f-a474-82e8e5c537f9-tmp\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnpv\" (UniqueName: \"kubernetes.io/projected/e5562a37-298a-406f-a474-82e8e5c537f9-kube-api-access-5nnpv\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.442601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.441713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5562a37-298a-406f-a474-82e8e5c537f9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.481176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.481132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.542480 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542480 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5562a37-298a-406f-a474-82e8e5c537f9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtsj\" (UniqueName: \"kubernetes.io/projected/4c3874e5-08a1-4b77-a197-1ab2a2f8d483-kube-api-access-pjtsj\") pod \"downloads-68c9cbb497-h8jff\" (UID: \"4c3874e5-08a1-4b77-a197-1ab2a2f8d483\") " pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c79e93-cb2e-4279-be58-a81d91e85f1e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bf908f-26ad-4edb-9b0b-7adf4008bd00-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542968 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7czf\" (UniqueName: \"kubernetes.io/projected/15c79e93-cb2e-4279-be58-a81d91e85f1e-kube-api-access-p7czf\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.542968 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbvd\" (UniqueName: \"kubernetes.io/projected/28bf908f-26ad-4edb-9b0b-7adf4008bd00-kube-api-access-rdbvd\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.542968 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5562a37-298a-406f-a474-82e8e5c537f9-tmp\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.542968 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.542815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnpv\" (UniqueName: \"kubernetes.io/projected/e5562a37-298a-406f-a474-82e8e5c537f9-kube-api-access-5nnpv\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.544076 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.544047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5562a37-298a-406f-a474-82e8e5c537f9-tmp\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.544583 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.544555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bf908f-26ad-4edb-9b0b-7adf4008bd00-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.545996 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.545969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-ca\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.546182 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.546127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.546811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.546786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5562a37-298a-406f-a474-82e8e5c537f9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.546896 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.546787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15c79e93-cb2e-4279-be58-a81d91e85f1e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.547658 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.547638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.547748 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.547676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/28bf908f-26ad-4edb-9b0b-7adf4008bd00-hub\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.554447 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.554380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnpv\" (UniqueName: \"kubernetes.io/projected/e5562a37-298a-406f-a474-82e8e5c537f9-kube-api-access-5nnpv\") pod \"klusterlet-addon-workmgr-5cfc565f97-sz9fs\" (UID: \"e5562a37-298a-406f-a474-82e8e5c537f9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.555426 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.555398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7czf\" (UniqueName: \"kubernetes.io/projected/15c79e93-cb2e-4279-be58-a81d91e85f1e-kube-api-access-p7czf\") pod \"managed-serviceaccount-addon-agent-f44585b44-4j6rt\" (UID: \"15c79e93-cb2e-4279-be58-a81d91e85f1e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.555988 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.555905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbvd\" (UniqueName: \"kubernetes.io/projected/28bf908f-26ad-4edb-9b0b-7adf4008bd00-kube-api-access-rdbvd\") pod \"cluster-proxy-proxy-agent-6d88fd865b-ts4cb\" (UID: \"28bf908f-26ad-4edb-9b0b-7adf4008bd00\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.576974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.576905 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.619694 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.619651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:18.633069 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.633037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" Jul 09 16:20:18.643684 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.643658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" Jul 09 16:20:18.643980 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.643947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtsj\" (UniqueName: \"kubernetes.io/projected/4c3874e5-08a1-4b77-a197-1ab2a2f8d483-kube-api-access-pjtsj\") pod \"downloads-68c9cbb497-h8jff\" (UID: \"4c3874e5-08a1-4b77-a197-1ab2a2f8d483\") " pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:18.659840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.659805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtsj\" (UniqueName: \"kubernetes.io/projected/4c3874e5-08a1-4b77-a197-1ab2a2f8d483-kube-api-access-pjtsj\") pod \"downloads-68c9cbb497-h8jff\" (UID: \"4c3874e5-08a1-4b77-a197-1ab2a2f8d483\") " pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:18.716044 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.716007 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:18.723034 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:18.722758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:20.039478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:20.039379 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:20.108793 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:20.108758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:20.667361 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:20.667321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:22.163181 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.163100 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb"] Jul 09 16:20:22.389888 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.389824 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt"] Jul 09 16:20:22.395166 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.394440 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs"] Jul 09 16:20:22.398072 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.397813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-h8jff"] Jul 09 16:20:22.402450 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:22.401345 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3874e5_08a1_4b77_a197_1ab2a2f8d483.slice/crio-25f9da3dae9ce08999c7bb8cc3bbc99ba3d268c1aac8e39402e017b9e370fd28 WatchSource:0}: Error finding container 25f9da3dae9ce08999c7bb8cc3bbc99ba3d268c1aac8e39402e017b9e370fd28: Status 404 returned error can't find the container with id 25f9da3dae9ce08999c7bb8cc3bbc99ba3d268c1aac8e39402e017b9e370fd28 Jul 09 16:20:22.402848 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:22.402800 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5562a37_298a_406f_a474_82e8e5c537f9.slice/crio-498e3a1dbcab742321cbe0427e256d505a3d836dc4891c910a47ad7d7a674aa7 WatchSource:0}: Error finding container 498e3a1dbcab742321cbe0427e256d505a3d836dc4891c910a47ad7d7a674aa7: Status 404 returned error can't find the container with id 498e3a1dbcab742321cbe0427e256d505a3d836dc4891c910a47ad7d7a674aa7 Jul 09 16:20:22.490183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.490142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfd8z" event={"ID":"9efaa331-652f-44ca-9f42-ba4a536de051","Type":"ContainerStarted","Data":"1055f16165bc849eb67c7522bf3c20efe52db276cdd08ebbc22c30aeb3ce537a"} Jul 09 16:20:22.490394 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.490235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zfd8z" event={"ID":"9efaa331-652f-44ca-9f42-ba4a536de051","Type":"ContainerStarted","Data":"35e14eae3927df840ef6b1401226d62699d77eaccb9600c8c0bdf6c462849580"} Jul 09 16:20:22.490394 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.490263 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zfd8z" Jul 09 16:20:22.492943 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.492500 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" event={"ID":"661c473a-c88b-430d-b27f-191bcb18a652","Type":"ContainerStarted","Data":"eece86c2e20f6881ae5b7ba4c81dc751607e29b72f6ae9fa0d8be5abdf574681"} Jul 09 16:20:22.495566 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.495532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" event={"ID":"28bf908f-26ad-4edb-9b0b-7adf4008bd00","Type":"ContainerStarted","Data":"8ba2caee70c10c255cfc5684ac2f81059422be3546401cb6c3a0fbeb6ddb093f"} Jul 09 16:20:22.496838 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.496815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" event={"ID":"e5562a37-298a-406f-a474-82e8e5c537f9","Type":"ContainerStarted","Data":"498e3a1dbcab742321cbe0427e256d505a3d836dc4891c910a47ad7d7a674aa7"} Jul 09 16:20:22.498130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.498110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" event={"ID":"15c79e93-cb2e-4279-be58-a81d91e85f1e","Type":"ContainerStarted","Data":"0c84f1ba4814000fcd6fd835e0a654df052cb8f3677f9734b05412e87c75ea22"} Jul 09 16:20:22.499167 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.499146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-h8jff" event={"ID":"4c3874e5-08a1-4b77-a197-1ab2a2f8d483","Type":"ContainerStarted","Data":"25f9da3dae9ce08999c7bb8cc3bbc99ba3d268c1aac8e39402e017b9e370fd28"} Jul 09 16:20:22.501193 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.501166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" event={"ID":"de8718da-a2ef-49b0-82ca-7f31e46519b4","Type":"ContainerStarted","Data":"e0807a5d0c5696b579acf58900d681719c92f06bea6d306ff2749e29fab4d2b0"} Jul 09 16:20:22.501277 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.501217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" event={"ID":"de8718da-a2ef-49b0-82ca-7f31e46519b4","Type":"ContainerStarted","Data":"7ca1b6b04043e50f0cd602947157812cc9e42150055d670c37410e253f0b4efb"} Jul 09 16:20:22.503391 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.503370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dffg" event={"ID":"5b6b02ae-6d00-43c6-b119-2ee2e78f0887","Type":"ContainerStarted","Data":"bd3accefe50f657b15517b7b11e5c5dd014b79dbfff2581e785e1ad205afb04d"} Jul 09 16:20:22.505216 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.505195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kr65g" event={"ID":"19baab78-2289-4c52-a951-febcda3d048a","Type":"ContainerStarted","Data":"cdeade91b44f58f28865a7bf7f47f19145bd148423ccddcec3be0ba26893aaa3"} Jul 09 16:20:22.513843 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.513792 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zfd8z" podStartSLOduration=34.612034973 podStartE2EDuration="41.51377438s" totalImagesPullingTime="6.901739407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:22.513227276 +0000 UTC m=+77.037551607" watchObservedRunningTime="2026-07-09 16:20:22.51377438 +0000 UTC m=+77.038098715" Jul 09 16:20:22.538964 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.538882 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9dffg" podStartSLOduration=17.105623065 podStartE2EDuration="28.538861582s" totalImagesPullingTime="11.433238517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:22.53840658 +0000 UTC m=+77.062730913" watchObservedRunningTime="2026-07-09 16:20:22.538861582 +0000 UTC m=+77.063185915" Jul 09 16:20:22.543893 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.543862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:22.557329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.557280 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-28vsp" podStartSLOduration=45.660290954 podStartE2EDuration="52.55726291s" totalImagesPullingTime="6.896971956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:22.556098647 +0000 UTC m=+77.080422991" watchObservedRunningTime="2026-07-09 16:20:22.55726291 +0000 UTC m=+77.081587243" Jul 09 16:20:22.572343 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.572295 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kr65g" podStartSLOduration=67.254653768 podStartE2EDuration="1m15.572282576s" totalImagesPullingTime="8.317628808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:22.571200329 +0000 UTC m=+77.095524662" watchObservedRunningTime="2026-07-09 16:20:22.572282576 +0000 UTC m=+77.096606908" Jul 09 16:20:22.588189 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.588147 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-nlql6" podStartSLOduration=45.632614264 podStartE2EDuration="52.588133117s" totalImagesPullingTime="6.955518853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:22.586945284 +0000 UTC m=+77.111269613" watchObservedRunningTime="2026-07-09 16:20:22.588133117 +0000 UTC m=+77.112457449" Jul 09 16:20:22.607287 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.607249 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:22.686225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.686178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:22.689863 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.689837 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww"] Jul 09 16:20:22.709146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.709110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww"] Jul 09 16:20:22.709303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.709217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:22.712300 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.712277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 16:20:22.712422 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.712341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tmjrj\"" Jul 09 16:20:22.881801 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.881699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpmww\" (UID: \"a135200a-fc55-4e49-beb5-61405c9132b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:22.982366 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:22.982326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpmww\" (UID: \"a135200a-fc55-4e49-beb5-61405c9132b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:22.982561 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:22.982489 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 16:20:22.982634 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:20:22.982569 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates podName:a135200a-fc55-4e49-beb5-61405c9132b4 nodeName:}" failed. No retries permitted until 2026-07-09 16:20:23.482548989 +0000 UTC m=+78.006873305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-dpmww" (UID: "a135200a-fc55-4e49-beb5-61405c9132b4") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 16:20:23.357358 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:23.357315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:23.487221 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:23.487182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpmww\" (UID: \"a135200a-fc55-4e49-beb5-61405c9132b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:23.493865 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:23.493832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a135200a-fc55-4e49-beb5-61405c9132b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpmww\" (UID: \"a135200a-fc55-4e49-beb5-61405c9132b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:23.619667 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:23.619588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:23.828454 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:23.828332 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww"] Jul 09 16:20:24.173596 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:24.173484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:24.273632 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:24.273533 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:24.522650 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:24.522476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" event={"ID":"a135200a-fc55-4e49-beb5-61405c9132b4","Type":"ContainerStarted","Data":"2e6fe423e5c0866e6b33edd44f25ddde3685d5f910c7398ac16701a42fdd427a"} Jul 09 16:20:26.098746 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:26.098638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:26.199748 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:26.199684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:30.560106 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.560064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" event={"ID":"28bf908f-26ad-4edb-9b0b-7adf4008bd00","Type":"ContainerStarted","Data":"9ded504ff4fb2a668b3d10dc4e925a4435e9a404c3102f06e3aaa053a5e65bad"} Jul 09 16:20:30.560535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.560123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" event={"ID":"28bf908f-26ad-4edb-9b0b-7adf4008bd00","Type":"ContainerStarted","Data":"c804b3b6e461a2c156579dcc1cdb65b02739d6efbe842a6bc63352d4f09f39d7"} Jul 09 16:20:30.560535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.560145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" event={"ID":"28bf908f-26ad-4edb-9b0b-7adf4008bd00","Type":"ContainerStarted","Data":"3db25ded43661c96214b4a464a8fdcc733e7515c3c1e471091748b9cd43aa342"} Jul 09 16:20:30.562084 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.562053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" event={"ID":"e5562a37-298a-406f-a474-82e8e5c537f9","Type":"ContainerStarted","Data":"1f8a7810cba4777cf99bbb0aa3ab054961a44f6fec397dee0bb46aa68532b408"} Jul 09 16:20:30.562240 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.562111 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:30.564181 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.564157 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" Jul 09 16:20:30.564428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.564403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" event={"ID":"15c79e93-cb2e-4279-be58-a81d91e85f1e","Type":"ContainerStarted","Data":"0ae6338d341ba8b7f465baf972f9811d1f1c12d924d9aa337727f6ea2b8c55e3"} Jul 09 16:20:30.566254 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.566229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" event={"ID":"a135200a-fc55-4e49-beb5-61405c9132b4","Type":"ContainerStarted","Data":"d8f86a0663c146d08cffdcbeb2dd3a922ccb19fe11ecb63ab04939a97660e149"} Jul 09 16:20:30.566453 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.566437 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:30.571488 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.571466 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" Jul 09 16:20:30.582037 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.581988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d88fd865b-ts4cb" podStartSLOduration=4.756545267 podStartE2EDuration="12.581969035s" totalImagesPullingTime="7.825423768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:30.579585438 +0000 UTC m=+85.103909771" watchObservedRunningTime="2026-07-09 16:20:30.581969035 +0000 UTC m=+85.106293368" Jul 09 16:20:30.597214 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.597161 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f44585b44-4j6rt" podStartSLOduration=5.010321271 podStartE2EDuration="12.597147568s" totalImagesPullingTime="7.586826297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:30.596022238 +0000 UTC m=+85.120346567" watchObservedRunningTime="2026-07-09 16:20:30.597147568 +0000 UTC m=+85.121471900" Jul 09 16:20:30.614132 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.614085 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpmww" podStartSLOduration=2.465650542 podStartE2EDuration="8.614069802s" totalImagesPullingTime="6.14841926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:30.613392994 +0000 UTC m=+85.137717328" watchObservedRunningTime="2026-07-09 16:20:30.614069802 +0000 UTC m=+85.138394133" Jul 09 16:20:30.633610 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.633554 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cfc565f97-sz9fs" podStartSLOduration=5.028498222 podStartE2EDuration="12.633532451s" totalImagesPullingTime="7.605034229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:30.631491095 +0000 UTC m=+85.155815429" watchObservedRunningTime="2026-07-09 16:20:30.633532451 +0000 UTC m=+85.157856784" Jul 09 16:20:30.714294 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.714258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:30.785751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:30.785718 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:32.000612 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:32.000561 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:32.100145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:32.100108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:32.164790 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:32.164758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:32.513829 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:32.513793 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zfd8z" Jul 09 16:20:33.661414 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:33.661380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:34.026042 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:34.026002 2577 patch_prober.go:28] interesting pod/image-registry-78858457bc-l66hw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 16:20:34.026225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:34.026063 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78858457bc-l66hw" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 16:20:34.073586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:34.073554 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:36.464655 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:36.464624 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:20:36.774857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:36.774820 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:36.831506 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:36.831468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:37.880797 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:37.880749 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:37.954511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:37.954482 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:38.131145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:38.131047 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:38.229087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:38.229055 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:39.177884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:39.177853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:39.242265 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:39.242226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:40.864459 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:40.864427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.077508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.077470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.135996 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.135900 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6857q"] Jul 09 16:20:41.174762 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.174605 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.274705 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.274663 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.375046 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.374515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.375752 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.375630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.377434 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.377410 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tv8r2\"" Jul 09 16:20:41.377540 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.377412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 16:20:41.378555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.378409 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 16:20:41.378555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.378412 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 16:20:41.378555 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.378487 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 16:20:41.446405 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-metrics-client-ca\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446405 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-root\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-sys\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-tls\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-textfile\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlg5m\" (UniqueName: \"kubernetes.io/projected/681bd3f6-5d93-46b8-8f7f-f8c906212618-kube-api-access-vlg5m\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446781 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-wtmp\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.446781 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.446616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-accelerators-collector-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.475244 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.475211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.547197 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-root\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-sys\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-root\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-sys\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-tls\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-textfile\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlg5m\" (UniqueName: \"kubernetes.io/projected/681bd3f6-5d93-46b8-8f7f-f8c906212618-kube-api-access-vlg5m\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547636 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-wtmp\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547636 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-accelerators-collector-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547636 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-metrics-client-ca\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547805 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-textfile\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.547905 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.547886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-wtmp\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.548151 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.548132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-metrics-client-ca\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.548320 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.548263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-accelerators-collector-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.550040 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.550021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.550360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.550337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/681bd3f6-5d93-46b8-8f7f-f8c906212618-node-exporter-tls\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.559704 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.559679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlg5m\" (UniqueName: \"kubernetes.io/projected/681bd3f6-5d93-46b8-8f7f-f8c906212618-kube-api-access-vlg5m\") pod \"node-exporter-6857q\" (UID: \"681bd3f6-5d93-46b8-8f7f-f8c906212618\") " pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.569225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.569199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.644714 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.644678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.685774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.685734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6857q" Jul 09 16:20:41.730344 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:20:41.730298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681bd3f6_5d93_46b8_8f7f_f8c906212618.slice/crio-6808c272bffc1d1169f1f9cf4b1828c4859ef0610dc8cd6bbf198d8dbf0488d5 WatchSource:0}: Error finding container 6808c272bffc1d1169f1f9cf4b1828c4859ef0610dc8cd6bbf198d8dbf0488d5: Status 404 returned error can't find the container with id 6808c272bffc1d1169f1f9cf4b1828c4859ef0610dc8cd6bbf198d8dbf0488d5 Jul 09 16:20:41.745964 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.745900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:41.830537 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:41.830505 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:42.103556 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.103521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:42.190236 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.190196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:42.613535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.613495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:42.624548 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.624508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6857q" event={"ID":"681bd3f6-5d93-46b8-8f7f-f8c906212618","Type":"ContainerStarted","Data":"6808c272bffc1d1169f1f9cf4b1828c4859ef0610dc8cd6bbf198d8dbf0488d5"} Jul 09 16:20:42.626558 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.626528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-h8jff" event={"ID":"4c3874e5-08a1-4b77-a197-1ab2a2f8d483","Type":"ContainerStarted","Data":"dcfb1bbf77697113d8b713e3bc1ea34d071645d2dd47e7f4eefaff46831c3de8"} Jul 09 16:20:42.627111 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.627084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:42.641753 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.641726 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-h8jff" Jul 09 16:20:42.646369 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:42.646316 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-h8jff" podStartSLOduration=5.278742854 podStartE2EDuration="24.646300848s" totalImagesPullingTime="19.367557994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:42.644086908 +0000 UTC m=+97.168411239" watchObservedRunningTime="2026-07-09 16:20:42.646300848 +0000 UTC m=+97.170625182" Jul 09 16:20:43.072525 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:43.072480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:43.139954 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:43.139894 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:43.421716 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:43.421632 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rd9v8" Jul 09 16:20:44.206358 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:44.206319 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:44.341430 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:44.341401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:45.602602 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.602557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:45.646394 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.646360 2577 generic.go:356] "Generic (PLEG): container finished" podID="681bd3f6-5d93-46b8-8f7f-f8c906212618" containerID="61b88ea3d6ecb8db73e71698b9758d27e07978004dd6e8cbedbd08e6596b7c2c" exitCode=0 Jul 09 16:20:45.646576 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.646452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6857q" event={"ID":"681bd3f6-5d93-46b8-8f7f-f8c906212618","Type":"ContainerDied","Data":"61b88ea3d6ecb8db73e71698b9758d27e07978004dd6e8cbedbd08e6596b7c2c"} Jul 09 16:20:45.673429 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.673398 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:45.894624 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.894397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:45.984807 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:45.984766 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:46.066343 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.066305 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:20:46.192108 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.191897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:46.291416 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.291369 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:46.394404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.394361 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:46.492639 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.492601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:46.654929 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.654878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6857q" event={"ID":"681bd3f6-5d93-46b8-8f7f-f8c906212618","Type":"ContainerStarted","Data":"e605a92364caa25518809357a9183565dbfe1cb9cb6dd04e0671c84037686734"} Jul 09 16:20:46.655392 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.655013 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6857q" event={"ID":"681bd3f6-5d93-46b8-8f7f-f8c906212618","Type":"ContainerStarted","Data":"d4ca95323481647891b2e1bcdb4ea40bfd7e25acd827d9b5e0c39cb0cf8b0cba"} Jul 09 16:20:46.677033 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:46.676977 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6857q" podStartSLOduration=2.7665017179999998 podStartE2EDuration="5.676954148s" totalImagesPullingTime="2.793234738s" totalInitContainerRuntime="117.217692ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:46.676312548 +0000 UTC m=+101.200636888" watchObservedRunningTime="2026-07-09 16:20:46.676954148 +0000 UTC m=+101.201278481" Jul 09 16:20:47.372736 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:47.372702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:47.499111 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:47.499073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:47.705390 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:47.705296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:47.802508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:47.802469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:47.902765 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:47.902728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:49.692570 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:49.692529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:49.792900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:49.792860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:51.139780 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:51.139719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:52.253337 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:52.253305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:52.355247 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:52.355212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:54.737489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:54.737454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:54.839987 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:54.839945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:58.319234 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:58.319197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:58.413558 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:58.413521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:20:58.477032 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:20:58.476989 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:00.165501 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:00.165462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:00.247960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:00.247908 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:00.360897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:00.360862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:00.448267 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:00.448166 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:04.768214 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:04.768180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:04.834430 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:04.834395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:06.386837 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:06.386796 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:21:08.748123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:08.748085 2577 generic.go:356] "Generic (PLEG): container finished" podID="edad3ea9-8b2c-4199-be52-06f37aaa936b" containerID="f4b48d3d78130f18544ae349af0176675a151c0fee23e13eeddbe4a0d63f073a" exitCode=0 Jul 09 16:21:08.748553 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:08.748156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" event={"ID":"edad3ea9-8b2c-4199-be52-06f37aaa936b","Type":"ContainerDied","Data":"f4b48d3d78130f18544ae349af0176675a151c0fee23e13eeddbe4a0d63f073a"} Jul 09 16:21:08.748553 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:08.748499 2577 scope.go:117] "RemoveContainer" containerID="f4b48d3d78130f18544ae349af0176675a151c0fee23e13eeddbe4a0d63f073a" Jul 09 16:21:09.753587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:09.753553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gp6cl" event={"ID":"edad3ea9-8b2c-4199-be52-06f37aaa936b","Type":"ContainerStarted","Data":"dd5534b159c26d440b47de36b908b6cfe5d0d5bf5fa392ddc4cea49c40d50f3f"} Jul 09 16:21:11.096476 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.096418 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78858457bc-l66hw" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerName="registry" containerID="cri-o://63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632" gracePeriod=30 Jul 09 16:21:11.381512 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.381483 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:21:11.525705 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525676 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.525886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.525886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525753 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.525886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.525886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525839 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.525886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525876 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.526177 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525912 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbcl2\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.526177 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.525985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets\") pod \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\" (UID: \"7cfe4a52-2ba4-4d2c-b50f-100866fd3820\") " Jul 09 16:21:11.526329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.526288 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:21:11.526424 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.526392 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:21:11.528806 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.528779 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:21:11.528939 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.528795 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:21:11.528939 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.528900 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2" (OuterVolumeSpecName: "kube-api-access-mbcl2") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "kube-api-access-mbcl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:21:11.529052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.528960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:21:11.529052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.529000 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:21:11.534285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.534254 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7cfe4a52-2ba4-4d2c-b50f-100866fd3820" (UID: "7cfe4a52-2ba4-4d2c-b50f-100866fd3820"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627120 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-certificates\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627152 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-registry-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627161 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-bound-sa-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627171 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-image-registry-private-configuration\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627182 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-trusted-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627189 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-ca-trust-extracted\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627198 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbcl2\" (UniqueName: \"kubernetes.io/projected/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-kube-api-access-mbcl2\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.627202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.627207 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7cfe4a52-2ba4-4d2c-b50f-100866fd3820-installation-pull-secrets\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:21:11.761832 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.761793 2577 generic.go:356] "Generic (PLEG): container finished" podID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerID="63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632" exitCode=0 Jul 09 16:21:11.761999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.761843 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78858457bc-l66hw" Jul 09 16:21:11.761999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.761859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78858457bc-l66hw" event={"ID":"7cfe4a52-2ba4-4d2c-b50f-100866fd3820","Type":"ContainerDied","Data":"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632"} Jul 09 16:21:11.761999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.761910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78858457bc-l66hw" event={"ID":"7cfe4a52-2ba4-4d2c-b50f-100866fd3820","Type":"ContainerDied","Data":"724e67a992bc19a1d1923b1258c298984f51239d3898d57fcee7940807f343d8"} Jul 09 16:21:11.761999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.761966 2577 scope.go:117] "RemoveContainer" containerID="63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632" Jul 09 16:21:11.771860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.771840 2577 scope.go:117] "RemoveContainer" containerID="63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632" Jul 09 16:21:11.772198 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:21:11.772177 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632\": container with ID starting with 63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632 not found: ID does not exist" containerID="63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632" Jul 09 16:21:11.772248 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.772209 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632"} err="failed to get container status \"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632\": rpc error: code = NotFound desc = could not find container \"63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632\": container with ID starting with 63989c6562e09cca7b8e68990ad7fa2bf0ed5f7d0006a8f46d56e03510944632 not found: ID does not exist" Jul 09 16:21:11.784474 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.784446 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:21:11.788567 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.788541 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78858457bc-l66hw"] Jul 09 16:21:11.992673 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:11.992638 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" path="/var/lib/kubelet/pods/7cfe4a52-2ba4-4d2c-b50f-100866fd3820/volumes" Jul 09 16:21:13.772789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.772757 2577 generic.go:356] "Generic (PLEG): container finished" podID="c71ed983-0744-4a12-9b54-fd0a633b24d9" containerID="77df450f40b2559f7622811d1cba8fa2979c2d2283da8d8fc912ce8c0b41fdaa" exitCode=0 Jul 09 16:21:13.773210 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.772828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" event={"ID":"c71ed983-0744-4a12-9b54-fd0a633b24d9","Type":"ContainerDied","Data":"77df450f40b2559f7622811d1cba8fa2979c2d2283da8d8fc912ce8c0b41fdaa"} Jul 09 16:21:13.773210 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.773186 2577 scope.go:117] "RemoveContainer" containerID="77df450f40b2559f7622811d1cba8fa2979c2d2283da8d8fc912ce8c0b41fdaa" Jul 09 16:21:13.774460 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.774433 2577 generic.go:356] "Generic (PLEG): container finished" podID="b8b10949-e84e-43ef-b4f2-4c900c949320" containerID="61705161fbb16fe2c15b9caeffd6fbbd5cad36eed9d37927767c91596138fa0d" exitCode=0 Jul 09 16:21:13.774552 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.774488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" event={"ID":"b8b10949-e84e-43ef-b4f2-4c900c949320","Type":"ContainerDied","Data":"61705161fbb16fe2c15b9caeffd6fbbd5cad36eed9d37927767c91596138fa0d"} Jul 09 16:21:13.774783 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:13.774768 2577 scope.go:117] "RemoveContainer" containerID="61705161fbb16fe2c15b9caeffd6fbbd5cad36eed9d37927767c91596138fa0d" Jul 09 16:21:14.780595 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:14.780554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-kgtns" event={"ID":"c71ed983-0744-4a12-9b54-fd0a633b24d9","Type":"ContainerStarted","Data":"4e22237de015ffe99f5142e7fbea6b224389f90990ec0fe944927e81bedcaf21"} Jul 09 16:21:14.782299 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:21:14.782276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cdlzl" event={"ID":"b8b10949-e84e-43ef-b4f2-4c900c949320","Type":"ContainerStarted","Data":"4920b49822a326237fa54e0819786dc784a2b7f06b014341922ec69cbfca00f1"} Jul 09 16:22:00.410222 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:00.410132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:00.510182 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:00.510146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:00.822605 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:00.822573 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:10.733986 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:10.733944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:21.712773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:21.712736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:21.821509 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:21.821475 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:21.887146 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:21.887107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:44.620843 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:44.620793 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:22:44.686148 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:22:44.686087 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:15.778735 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:15.778694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:15.835574 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:15.835543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:17.092474 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:17.092427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:17.191869 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:17.191824 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:27.778717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:27.778639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:28.000243 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:28.000207 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:28.172360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:28.172276 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:34.704439 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:34.704392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:23:35.629526 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:23:35.629481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:24:05.883634 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:05.883602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:24:05.892025 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:05.892001 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:24:05.893059 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:05.893034 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:24:05.895849 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:05.895831 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 16:24:05.901478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:05.901460 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:24:38.172048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:38.172002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:24:40.737070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:40.737037 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:24:56.377609 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:56.377564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:24:56.468906 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:24:56.468873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:18.181910 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:18.181861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:18.975597 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:18.975559 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:21.807501 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:21.807460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:21.905860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:21.905829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:23.449279 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:23.449236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:23.561562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:23.561518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:23.681601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:23.681565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:24.655417 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:24.655370 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:24.756283 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:24.756227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:25.843118 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:25.843059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:26.264715 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:26.264680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:28.592698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:28.592661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:28.696445 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:28.696403 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:33.054426 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:33.054383 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:34.802391 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:34.802358 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:34.900126 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:34.900090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:39.986981 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:39.986948 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:40.089778 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:40.089741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:42.156835 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:42.156795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:50.118161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:50.118114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:50.215812 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:50.215768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:59.213216 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:59.213173 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:59.312505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:59.312462 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:25:59.816933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:25:59.816881 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.034446 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.034407 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.141643 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.141545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.207863 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.207833 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j"] Jul 09 16:26:00.208167 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.208155 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerName="registry" Jul 09 16:26:00.208220 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.208169 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerName="registry" Jul 09 16:26:00.208257 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.208223 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfe4a52-2ba4-4d2c-b50f-100866fd3820" containerName="registry" Jul 09 16:26:00.211235 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.211219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.215890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.215861 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 16:26:00.216814 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.216795 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 16:26:00.217519 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.217338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 16:26:00.220672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.220647 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kb6qk\"" Jul 09 16:26:00.220772 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.220695 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 16:26:00.236513 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.236485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j"] Jul 09 16:26:00.241037 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.241009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsk7\" (UniqueName: \"kubernetes.io/projected/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-kube-api-access-dbsk7\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.241147 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.241059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-tmp\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.241147 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.241130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.241230 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.241211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-manifests\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.241265 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.241246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342020 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.341977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsk7\" (UniqueName: \"kubernetes.io/projected/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-kube-api-access-dbsk7\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342020 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.342024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-tmp\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342249 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.342071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342249 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.342096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-manifests\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342249 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.342117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.342576 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.342549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-manifests\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.344662 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.344629 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-tmp\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.344858 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.344842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.344902 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.344891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.351766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.351740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsk7\" (UniqueName: \"kubernetes.io/projected/18b8ed4e-e86e-466a-b06e-46b52cb3fe92-kube-api-access-dbsk7\") pod \"opendatahub-operator-controller-manager-5bd696597f-vp54j\" (UID: \"18b8ed4e-e86e-466a-b06e-46b52cb3fe92\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.387731 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.387687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.495791 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.495761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.521699 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.521667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:00.590004 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.589943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.674507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.674442 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j"] Jul 09 16:26:00.677709 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.677679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.679002 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:26:00.678972 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b8ed4e_e86e_466a_b06e_46b52cb3fe92.slice/crio-89ad7198b60c3a515097862f5f3e30e3d01187d022153233c1338670dfd5c723 WatchSource:0}: Error finding container 89ad7198b60c3a515097862f5f3e30e3d01187d022153233c1338670dfd5c723: Status 404 returned error can't find the container with id 89ad7198b60c3a515097862f5f3e30e3d01187d022153233c1338670dfd5c723 Jul 09 16:26:00.680742 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.680726 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:26:00.755389 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.755290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.853029 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.852994 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:00.910350 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:00.910318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" event={"ID":"18b8ed4e-e86e-466a-b06e-46b52cb3fe92","Type":"ContainerStarted","Data":"89ad7198b60c3a515097862f5f3e30e3d01187d022153233c1338670dfd5c723"} Jul 09 16:26:04.465900 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:04.465842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:04.565774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:04.565662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:06.944048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:06.944015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" event={"ID":"18b8ed4e-e86e-466a-b06e-46b52cb3fe92","Type":"ContainerStarted","Data":"b7ce8914819050451ddb62f1189f5bdf0f27b2ee6c041f32abc8f10ab2fdc62c"} Jul 09 16:26:07.949390 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:07.949355 2577 generic.go:356] "Generic (PLEG): container finished" podID="18b8ed4e-e86e-466a-b06e-46b52cb3fe92" containerID="b7ce8914819050451ddb62f1189f5bdf0f27b2ee6c041f32abc8f10ab2fdc62c" exitCode=0 Jul 09 16:26:07.949823 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:07.949446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" event={"ID":"18b8ed4e-e86e-466a-b06e-46b52cb3fe92","Type":"ContainerDied","Data":"b7ce8914819050451ddb62f1189f5bdf0f27b2ee6c041f32abc8f10ab2fdc62c"} Jul 09 16:26:08.956033 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:08.955991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" event={"ID":"18b8ed4e-e86e-466a-b06e-46b52cb3fe92","Type":"ContainerStarted","Data":"00f1045b3474a358c5af58abe1aca42785a2005a8116fcbcd500ad86e4e26859"} Jul 09 16:26:08.956456 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:08.956130 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:08.983939 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:08.983851 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" podStartSLOduration=2.36123256 podStartE2EDuration="8.98383073s" totalImagesPullingTime="5.579299682s" totalInitContainerRuntime="1.043298488s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:08.98342811 +0000 UTC m=+423.507752442" watchObservedRunningTime="2026-07-09 16:26:08.98383073 +0000 UTC m=+423.508155096" Jul 09 16:26:10.515972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:10.515911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:10.614293 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:10.614247 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:12.098508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:12.098476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:12.205245 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:12.205210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:19.962413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:19.962384 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bd696597f-vp54j" Jul 09 16:26:21.754576 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:21.754519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:22.051224 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:22.051145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:22.639142 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:22.639095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:22.780334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:22.780294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:22.909356 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:22.909268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:23.054836 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:23.054789 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:23.592715 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:23.592679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:23.958312 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:23.958205 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:28.053734 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:28.053596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:28.209980 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:28.209936 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:29.698534 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.698502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:29.766044 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.766013 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s"] Jul 09 16:26:29.769312 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.769282 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776002 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776114 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4gfbp\"" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776220 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776261 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776307 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 16:26:29.776379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.776312 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 16:26:29.785897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.785875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s"] Jul 09 16:26:29.806953 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.806911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:29.899745 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.899712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-metrics-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:29.899745 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.899747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-manager-config\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:29.899987 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.899771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:29.899987 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:29.899802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x6pr\" (UniqueName: \"kubernetes.io/projected/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-kube-api-access-5x6pr\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.000467 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.000427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x6pr\" (UniqueName: \"kubernetes.io/projected/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-kube-api-access-5x6pr\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.000665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.000523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-metrics-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.000665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.000551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-manager-config\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.000665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.000580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.001446 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.001415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-manager-config\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.003319 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.003294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-metrics-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.003423 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.003329 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-cert\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.018313 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.018281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x6pr\" (UniqueName: \"kubernetes.io/projected/7f40f9f0-d0a3-468c-a43a-7a7aad57dd95-kube-api-access-5x6pr\") pod \"lws-controller-manager-7d5799cc79-b2v7s\" (UID: \"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95\") " pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.078996 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.078961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:30.221230 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:30.221196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s"] Jul 09 16:26:30.225104 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:26:30.225067 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f40f9f0_d0a3_468c_a43a_7a7aad57dd95.slice/crio-87ec82c521fc48a383a8f55c4a1f304383d69482f357210b3c42725730a520cb WatchSource:0}: Error finding container 87ec82c521fc48a383a8f55c4a1f304383d69482f357210b3c42725730a520cb: Status 404 returned error can't find the container with id 87ec82c521fc48a383a8f55c4a1f304383d69482f357210b3c42725730a520cb Jul 09 16:26:31.047881 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:31.047838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" event={"ID":"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95","Type":"ContainerStarted","Data":"87ec82c521fc48a383a8f55c4a1f304383d69482f357210b3c42725730a520cb"} Jul 09 16:26:31.643601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:31.643560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:31.772433 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:31.772396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:33.040410 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:33.040357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:33.216585 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:33.216538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:33.392830 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:33.392715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:33.506149 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:33.506106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:33.603413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:33.603374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:34.472060 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:34.472024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:34.576544 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:34.576509 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:34.701831 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:34.701797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:36.759656 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:36.759615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:37.140621 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:37.140524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:37.333665 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:37.333630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:26:52.145725 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:52.145686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" event={"ID":"7f40f9f0-d0a3-468c-a43a-7a7aad57dd95","Type":"ContainerStarted","Data":"d465ca91a6d7a61338893beec653ffe0e356caad7bdddedad16febe005523295"} Jul 09 16:26:52.146165 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:52.145772 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:26:52.163123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:52.163065 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" podStartSLOduration=2.026205543 podStartE2EDuration="23.163050084s" totalImagesPullingTime="21.136844541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:52.161946565 +0000 UTC m=+466.686270896" watchObservedRunningTime="2026-07-09 16:26:52.163050084 +0000 UTC m=+466.687374488" Jul 09 16:26:52.501424 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:26:52.501391 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:03.152043 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:03.152009 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d5799cc79-b2v7s" Jul 09 16:27:16.276722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:16.276681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:23.263857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.263818 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:23.361651 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.361616 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:23.388093 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.388057 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:23.396568 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.396546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:23.402194 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.402165 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wzdrc\"" Jul 09 16:27:23.402333 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.402225 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 16:27:23.402603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.402573 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 16:27:23.403252 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.403227 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:23.462134 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.462099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:23.546193 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.546088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qsb\" (UniqueName: \"kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb\") pod \"kuadrant-operator-catalog-ffqrt\" (UID: \"d5e6e5f3-c40b-4867-aed6-964b22ee940b\") " pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:23.647343 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.647304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qsb\" (UniqueName: \"kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb\") pod \"kuadrant-operator-catalog-ffqrt\" (UID: \"d5e6e5f3-c40b-4867-aed6-964b22ee940b\") " pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:23.658416 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.658376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qsb\" (UniqueName: \"kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb\") pod \"kuadrant-operator-catalog-ffqrt\" (UID: \"d5e6e5f3-c40b-4867-aed6-964b22ee940b\") " pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:23.708462 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.708422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:23.747159 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.747120 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:23.846227 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.846171 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:23.848722 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:27:23.848691 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e6e5f3_c40b_4867_aed6_964b22ee940b.slice/crio-a199156591caf011ba2abe7ad9c78d55f4e673c56b2797e0c6ab2f6010f2fef4 WatchSource:0}: Error finding container a199156591caf011ba2abe7ad9c78d55f4e673c56b2797e0c6ab2f6010f2fef4: Status 404 returned error can't find the container with id a199156591caf011ba2abe7ad9c78d55f4e673c56b2797e0c6ab2f6010f2fef4 Jul 09 16:27:23.957694 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.957664 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n6svw"] Jul 09 16:27:23.962876 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.962857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:23.970746 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:23.970720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n6svw"] Jul 09 16:27:24.050425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.050391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbknn\" (UniqueName: \"kubernetes.io/projected/ae95a51b-6238-4d90-8e10-02741e66c86c-kube-api-access-dbknn\") pod \"kuadrant-operator-catalog-n6svw\" (UID: \"ae95a51b-6238-4d90-8e10-02741e66c86c\") " pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:24.151268 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.151173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbknn\" (UniqueName: \"kubernetes.io/projected/ae95a51b-6238-4d90-8e10-02741e66c86c-kube-api-access-dbknn\") pod \"kuadrant-operator-catalog-n6svw\" (UID: \"ae95a51b-6238-4d90-8e10-02741e66c86c\") " pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:24.160250 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.160222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbknn\" (UniqueName: \"kubernetes.io/projected/ae95a51b-6238-4d90-8e10-02741e66c86c-kube-api-access-dbknn\") pod \"kuadrant-operator-catalog-n6svw\" (UID: \"ae95a51b-6238-4d90-8e10-02741e66c86c\") " pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:24.182020 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.181983 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:24.273198 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.273164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:24.282730 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.282702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" event={"ID":"d5e6e5f3-c40b-4867-aed6-964b22ee940b","Type":"ContainerStarted","Data":"a199156591caf011ba2abe7ad9c78d55f4e673c56b2797e0c6ab2f6010f2fef4"} Jul 09 16:27:24.389072 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.389034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:24.420783 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:24.420707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n6svw"] Jul 09 16:27:24.423714 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:27:24.423690 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae95a51b_6238_4d90_8e10_02741e66c86c.slice/crio-2b469b67c5a91e7f6c0b36e64e917dc046e6f5d4f97d13f0014f44e3c2177c83 WatchSource:0}: Error finding container 2b469b67c5a91e7f6c0b36e64e917dc046e6f5d4f97d13f0014f44e3c2177c83: Status 404 returned error can't find the container with id 2b469b67c5a91e7f6c0b36e64e917dc046e6f5d4f97d13f0014f44e3c2177c83 Jul 09 16:27:25.290199 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:25.290155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" event={"ID":"ae95a51b-6238-4d90-8e10-02741e66c86c","Type":"ContainerStarted","Data":"2b469b67c5a91e7f6c0b36e64e917dc046e6f5d4f97d13f0014f44e3c2177c83"} Jul 09 16:27:26.295790 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.295749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" event={"ID":"d5e6e5f3-c40b-4867-aed6-964b22ee940b","Type":"ContainerStarted","Data":"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825"} Jul 09 16:27:26.296275 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.295822 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" podUID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" containerName="registry-server" containerID="cri-o://fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825" gracePeriod=2 Jul 09 16:27:26.297353 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.297323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" event={"ID":"ae95a51b-6238-4d90-8e10-02741e66c86c","Type":"ContainerStarted","Data":"07b53e1d44927e541f133e43d8af53b7c1b3c77abaa4509b45fcc76a3adc3e1f"} Jul 09 16:27:26.315345 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.315290 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" podStartSLOduration=1.193361519 podStartE2EDuration="3.315268579s" totalImagesPullingTime="2.12190706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:27:26.313881863 +0000 UTC m=+500.838206202" watchObservedRunningTime="2026-07-09 16:27:26.315268579 +0000 UTC m=+500.839592911" Jul 09 16:27:26.330586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.330532 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" podStartSLOduration=1.782804982 podStartE2EDuration="3.33051679s" totalImagesPullingTime="1.547711808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:27:26.329038003 +0000 UTC m=+500.853362349" watchObservedRunningTime="2026-07-09 16:27:26.33051679 +0000 UTC m=+500.854841121" Jul 09 16:27:26.539963 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.539930 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:26.674840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.674751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qsb\" (UniqueName: \"kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb\") pod \"d5e6e5f3-c40b-4867-aed6-964b22ee940b\" (UID: \"d5e6e5f3-c40b-4867-aed6-964b22ee940b\") " Jul 09 16:27:26.677245 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.677216 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb" (OuterVolumeSpecName: "kube-api-access-75qsb") pod "d5e6e5f3-c40b-4867-aed6-964b22ee940b" (UID: "d5e6e5f3-c40b-4867-aed6-964b22ee940b"). InnerVolumeSpecName "kube-api-access-75qsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:27:26.776081 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.776045 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qsb\" (UniqueName: \"kubernetes.io/projected/d5e6e5f3-c40b-4867-aed6-964b22ee940b-kube-api-access-75qsb\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:27:26.921142 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:26.921106 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:27.023492 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.023454 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:27.122650 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.122611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:27.302966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.302853 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" containerID="fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825" exitCode=0 Jul 09 16:27:27.302966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.302896 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" Jul 09 16:27:27.302966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.302943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" event={"ID":"d5e6e5f3-c40b-4867-aed6-964b22ee940b","Type":"ContainerDied","Data":"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825"} Jul 09 16:27:27.303522 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.302996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ffqrt" event={"ID":"d5e6e5f3-c40b-4867-aed6-964b22ee940b","Type":"ContainerDied","Data":"a199156591caf011ba2abe7ad9c78d55f4e673c56b2797e0c6ab2f6010f2fef4"} Jul 09 16:27:27.303522 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.303017 2577 scope.go:117] "RemoveContainer" containerID="fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825" Jul 09 16:27:27.312207 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.312187 2577 scope.go:117] "RemoveContainer" containerID="fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825" Jul 09 16:27:27.312497 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:27:27.312478 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825\": container with ID starting with fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825 not found: ID does not exist" containerID="fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825" Jul 09 16:27:27.312573 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.312511 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825"} err="failed to get container status \"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825\": rpc error: code = NotFound desc = could not find container \"fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825\": container with ID starting with fc140092a000f0747ca0783438bd564a18262647ff27106dd39ee3d92d41e825 not found: ID does not exist" Jul 09 16:27:27.325621 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.325589 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:27.327884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.327860 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ffqrt"] Jul 09 16:27:27.992304 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:27.992271 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" path="/var/lib/kubelet/pods/d5e6e5f3-c40b-4867-aed6-964b22ee940b/volumes" Jul 09 16:27:34.273446 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:34.273408 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:34.273842 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:34.273465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:34.296365 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:34.296329 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:34.357274 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:34.357243 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-n6svw" Jul 09 16:27:49.442642 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:49.442609 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:52.893395 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:52.893366 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:54.854228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:54.854198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:56.245229 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.245190 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht"] Jul 09 16:27:56.245693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.245647 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" containerName="registry-server" Jul 09 16:27:56.245693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.245666 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" containerName="registry-server" Jul 09 16:27:56.245802 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.245783 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e6e5f3-c40b-4867-aed6-964b22ee940b" containerName="registry-server" Jul 09 16:27:56.250258 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.250233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:56.257226 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.257205 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mlk2r\"" Jul 09 16:27:56.257327 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.257267 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 16:27:56.267176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.267144 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht"] Jul 09 16:27:56.314028 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.313987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgdk\" (UniqueName: \"kubernetes.io/projected/877460aa-b6be-4672-bb93-5a296a103fee-kube-api-access-xsgdk\") pod \"dns-operator-controller-manager-864dd78c4c-xbdht\" (UID: \"877460aa-b6be-4672-bb93-5a296a103fee\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:56.414905 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.414864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgdk\" (UniqueName: \"kubernetes.io/projected/877460aa-b6be-4672-bb93-5a296a103fee-kube-api-access-xsgdk\") pod \"dns-operator-controller-manager-864dd78c4c-xbdht\" (UID: \"877460aa-b6be-4672-bb93-5a296a103fee\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:56.424944 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.424902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgdk\" (UniqueName: \"kubernetes.io/projected/877460aa-b6be-4672-bb93-5a296a103fee-kube-api-access-xsgdk\") pod \"dns-operator-controller-manager-864dd78c4c-xbdht\" (UID: \"877460aa-b6be-4672-bb93-5a296a103fee\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:56.505542 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.505452 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:56.560202 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.560164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:56.701367 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:56.701344 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht"] Jul 09 16:27:56.704328 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:27:56.704292 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877460aa_b6be_4672_bb93_5a296a103fee.slice/crio-c8a70cf84e110b8e900c96b237367094718a159bd439a94d15a6b76cb6a760aa WatchSource:0}: Error finding container c8a70cf84e110b8e900c96b237367094718a159bd439a94d15a6b76cb6a760aa: Status 404 returned error can't find the container with id c8a70cf84e110b8e900c96b237367094718a159bd439a94d15a6b76cb6a760aa Jul 09 16:27:57.437993 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:57.437958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" event={"ID":"877460aa-b6be-4672-bb93-5a296a103fee","Type":"ContainerStarted","Data":"c8a70cf84e110b8e900c96b237367094718a159bd439a94d15a6b76cb6a760aa"} Jul 09 16:27:57.450047 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:57.450020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:58.886386 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:58.886349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:59.290312 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.290273 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk"] Jul 09 16:27:59.293790 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.293765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:27:59.297012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.296989 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gsbl2\"" Jul 09 16:27:59.310119 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.310094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk"] Jul 09 16:27:59.337116 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.337073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgfb\" (UniqueName: \"kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb\") pod \"limitador-operator-controller-manager-859d7b78bd-q8jmk\" (UID: \"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:27:59.437880 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.437765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgfb\" (UniqueName: \"kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb\") pod \"limitador-operator-controller-manager-859d7b78bd-q8jmk\" (UID: \"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:27:59.448093 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.448058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" event={"ID":"877460aa-b6be-4672-bb93-5a296a103fee","Type":"ContainerStarted","Data":"7feef1b043f6e37061b73629353cbb9c93c8fa070d6f758c9a965e42d88a2e58"} Jul 09 16:27:59.448240 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.448190 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:27:59.450280 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.450260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgfb\" (UniqueName: \"kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb\") pod \"limitador-operator-controller-manager-859d7b78bd-q8jmk\" (UID: \"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:27:59.468975 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.468903 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" podStartSLOduration=1.066980052 podStartE2EDuration="3.468888445s" totalImagesPullingTime="2.401908393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:27:59.467758235 +0000 UTC m=+533.992082566" watchObservedRunningTime="2026-07-09 16:27:59.468888445 +0000 UTC m=+533.993212777" Jul 09 16:27:59.603768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.603727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:27:59.650440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.650401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:27:59.745796 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:27:59.745766 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk"] Jul 09 16:27:59.748766 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:27:59.748741 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba4fbd1_72ae_44ba_bff0_b5f7744bcdb9.slice/crio-60c9a4f03ffabce6bcd9162e49afab0311768f40b9cd81f11dd463432660ad6c WatchSource:0}: Error finding container 60c9a4f03ffabce6bcd9162e49afab0311768f40b9cd81f11dd463432660ad6c: Status 404 returned error can't find the container with id 60c9a4f03ffabce6bcd9162e49afab0311768f40b9cd81f11dd463432660ad6c Jul 09 16:28:00.245094 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:00.245053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:00.458092 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:00.458059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" event={"ID":"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9","Type":"ContainerStarted","Data":"60c9a4f03ffabce6bcd9162e49afab0311768f40b9cd81f11dd463432660ad6c"} Jul 09 16:28:01.220908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:01.220873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:01.320689 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:01.320638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:02.469510 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.469474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" event={"ID":"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9","Type":"ContainerStarted","Data":"7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8"} Jul 09 16:28:02.470012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.469544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:02.490382 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.490331 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" podStartSLOduration=1.693194188 podStartE2EDuration="3.490312574s" totalImagesPullingTime="1.797118386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:28:02.488394531 +0000 UTC m=+537.012718876" watchObservedRunningTime="2026-07-09 16:28:02.490312574 +0000 UTC m=+537.014636906" Jul 09 16:28:02.581231 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.581194 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf"] Jul 09 16:28:02.584524 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.584507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:02.587662 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.587645 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-d8qww\"" Jul 09 16:28:02.595318 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.595297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf"] Jul 09 16:28:02.664294 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.664242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblg9\" (UniqueName: \"kubernetes.io/projected/5e035f87-0c3b-41b2-8cc0-ae55c2f14c08-kube-api-access-kblg9\") pod \"authorino-operator-75cc8d5cb7-x5tpf\" (UID: \"5e035f87-0c3b-41b2-8cc0-ae55c2f14c08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:02.765425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.765381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblg9\" (UniqueName: \"kubernetes.io/projected/5e035f87-0c3b-41b2-8cc0-ae55c2f14c08-kube-api-access-kblg9\") pod \"authorino-operator-75cc8d5cb7-x5tpf\" (UID: \"5e035f87-0c3b-41b2-8cc0-ae55c2f14c08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:02.774621 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.774597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblg9\" (UniqueName: \"kubernetes.io/projected/5e035f87-0c3b-41b2-8cc0-ae55c2f14c08-kube-api-access-kblg9\") pod \"authorino-operator-75cc8d5cb7-x5tpf\" (UID: \"5e035f87-0c3b-41b2-8cc0-ae55c2f14c08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:02.895702 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:02.895658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:03.039911 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:03.039790 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf"] Jul 09 16:28:03.042954 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:28:03.042909 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e035f87_0c3b_41b2_8cc0_ae55c2f14c08.slice/crio-ec4db4e935d236d993bf38b9c7378d92b16bd5a6c569ca730ea9372e11dc2417 WatchSource:0}: Error finding container ec4db4e935d236d993bf38b9c7378d92b16bd5a6c569ca730ea9372e11dc2417: Status 404 returned error can't find the container with id ec4db4e935d236d993bf38b9c7378d92b16bd5a6c569ca730ea9372e11dc2417 Jul 09 16:28:03.130580 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:03.130545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:03.331640 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:03.331548 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:03.475519 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:03.475485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" event={"ID":"5e035f87-0c3b-41b2-8cc0-ae55c2f14c08","Type":"ContainerStarted","Data":"ec4db4e935d236d993bf38b9c7378d92b16bd5a6c569ca730ea9372e11dc2417"} Jul 09 16:28:04.262731 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:04.262680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:04.362338 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:04.362306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:06.491755 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:06.491708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" event={"ID":"5e035f87-0c3b-41b2-8cc0-ae55c2f14c08","Type":"ContainerStarted","Data":"1f3aa8d64efe8364490e86a4d1ec7e3c8efd8c1beffa6baa3c18f56dc1851f00"} Jul 09 16:28:06.492186 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:06.491861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:06.509193 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:06.509133 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" podStartSLOduration=1.4316625570000001 podStartE2EDuration="4.509119344s" totalImagesPullingTime="3.077456787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:28:06.508294291 +0000 UTC m=+541.032618624" watchObservedRunningTime="2026-07-09 16:28:06.509119344 +0000 UTC m=+541.033443675" Jul 09 16:28:10.461061 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:10.461017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xbdht" Jul 09 16:28:13.086573 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.086526 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:13.184550 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.184519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:13.478002 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.477974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:13.626378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.626340 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk"] Jul 09 16:28:13.626622 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.626569 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" containerName="manager" containerID="cri-o://7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8" gracePeriod=2 Jul 09 16:28:13.648520 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:28:13.648479 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-q8jmk\": pods \"limitador-operator-controller-manager-859d7b78bd-q8jmk\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:13.649054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.649027 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk"] Jul 09 16:28:13.870909 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.870886 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:13.873686 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:28:13.873655 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8jmk\" is forbidden: User \"system:node:ip-10-0-133-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-162.ec2.internal' and this object" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:13.961170 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.961134 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slgfb\" (UniqueName: \"kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb\") pod \"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\" (UID: \"3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9\") " Jul 09 16:28:13.963461 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.963429 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb" (OuterVolumeSpecName: "kube-api-access-slgfb") pod "3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" (UID: "3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9"). InnerVolumeSpecName "kube-api-access-slgfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:28:13.993637 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:13.993563 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" path="/var/lib/kubelet/pods/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9/volumes" Jul 09 16:28:14.061888 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.061857 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgfb\" (UniqueName: \"kubernetes.io/projected/3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9-kube-api-access-slgfb\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:28:14.530630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.530599 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" containerID="7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8" exitCode=0 Jul 09 16:28:14.531070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.530652 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8jmk" Jul 09 16:28:14.531070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.530666 2577 scope.go:117] "RemoveContainer" containerID="7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8" Jul 09 16:28:14.539700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.539681 2577 scope.go:117] "RemoveContainer" containerID="7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8" Jul 09 16:28:14.540048 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:28:14.540026 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8\": container with ID starting with 7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8 not found: ID does not exist" containerID="7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8" Jul 09 16:28:14.540116 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:14.540057 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8"} err="failed to get container status \"7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8\": rpc error: code = NotFound desc = could not find container \"7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8\": container with ID starting with 7f7ad8882d2e5bd34253747564a212489d5257e05d0cfe38fd1fd008af3bb2a8 not found: ID does not exist" Jul 09 16:28:16.882234 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:16.876562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:16.975971 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:16.975910 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:17.102291 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:17.102252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:17.499023 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:17.498986 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-x5tpf" Jul 09 16:28:21.673320 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:21.673275 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:21.772428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:21.772396 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:25.662188 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:25.662145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:25.811452 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:25.811412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:33.656432 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:33.656395 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:33.718983 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:33.718909 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:37.869654 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:37.869613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:37.973467 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:37.973421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.263097 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.263066 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.322382 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.322343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.469981 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.469949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.568523 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.568427 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.663832 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.663798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:42.839147 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:42.839045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:45.219938 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.219893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:45.227155 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.227127 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:28:45.227462 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.227448 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" containerName="manager" Jul 09 16:28:45.227503 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.227464 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" containerName="manager" Jul 09 16:28:45.227536 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.227527 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba4fbd1-72ae-44ba-bff0-b5f7744bcdb9" containerName="manager" Jul 09 16:28:45.235550 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.235525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.238597 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.238549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 16:28:45.238852 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.238831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n7pqw\"" Jul 09 16:28:45.239237 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.239211 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:28:45.321648 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.321615 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:28:45.326827 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.326786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.326997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.326870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qsn\" (UniqueName: \"kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.336898 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.336857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:45.395396 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.395356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:45.427485 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.427446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qsn\" (UniqueName: \"kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.427674 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.427552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.428332 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.428311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.437097 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.437076 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qsn\" (UniqueName: \"kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn\") pod \"limitador-limitador-64c94fd64f-lwc4w\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.547632 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.547593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:45.685832 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:45.685728 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:28:45.688235 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:28:45.688200 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f25cca_a183_4e0e_bcb9_efc0c899f8f9.slice/crio-19428b1a1207fc4ad2511817a3a7d488a1401cbfcd15417ae27e5ca9125cf120 WatchSource:0}: Error finding container 19428b1a1207fc4ad2511817a3a7d488a1401cbfcd15417ae27e5ca9125cf120: Status 404 returned error can't find the container with id 19428b1a1207fc4ad2511817a3a7d488a1401cbfcd15417ae27e5ca9125cf120 Jul 09 16:28:46.088774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.088723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:46.165052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.165014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:28:46.182130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.182100 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:28:46.186813 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.186795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:28:46.189856 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.189830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5mhv5\"" Jul 09 16:28:46.191177 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.191152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:28:46.334776 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.334737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brs45\" (UniqueName: \"kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45\") pod \"authorino-75d556bd77-d9f7w\" (UID: \"bedb7f61-a918-44c7-bbe0-ee767b06f80e\") " pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:28:46.435746 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.435657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brs45\" (UniqueName: \"kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45\") pod \"authorino-75d556bd77-d9f7w\" (UID: \"bedb7f61-a918-44c7-bbe0-ee767b06f80e\") " pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:28:46.449046 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.448983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brs45\" (UniqueName: \"kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45\") pod \"authorino-75d556bd77-d9f7w\" (UID: \"bedb7f61-a918-44c7-bbe0-ee767b06f80e\") " pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:28:46.498385 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.498346 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:28:46.683774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.683737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" event={"ID":"66f25cca-a183-4e0e-bcb9-efc0c899f8f9","Type":"ContainerStarted","Data":"19428b1a1207fc4ad2511817a3a7d488a1401cbfcd15417ae27e5ca9125cf120"} Jul 09 16:28:46.691446 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:46.691418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:28:46.694534 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:28:46.694502 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedb7f61_a918_44c7_bbe0_ee767b06f80e.slice/crio-556989e6d9b0d5eacf2c49dd5fc826bf8e2edde0c9b82df3446041d8a3140287 WatchSource:0}: Error finding container 556989e6d9b0d5eacf2c49dd5fc826bf8e2edde0c9b82df3446041d8a3140287: Status 404 returned error can't find the container with id 556989e6d9b0d5eacf2c49dd5fc826bf8e2edde0c9b82df3446041d8a3140287 Jul 09 16:28:47.689999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:47.689960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d9f7w" event={"ID":"bedb7f61-a918-44c7-bbe0-ee767b06f80e","Type":"ContainerStarted","Data":"556989e6d9b0d5eacf2c49dd5fc826bf8e2edde0c9b82df3446041d8a3140287"} Jul 09 16:28:48.699389 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:48.699352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" event={"ID":"66f25cca-a183-4e0e-bcb9-efc0c899f8f9","Type":"ContainerStarted","Data":"9b53de0cfa8d2eb6bcb6b4a376c877bfbe22c3fd39b85495b5b194f9985b3c72"} Jul 09 16:28:48.699817 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:48.699428 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:28:51.714545 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:51.714505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d9f7w" event={"ID":"bedb7f61-a918-44c7-bbe0-ee767b06f80e","Type":"ContainerStarted","Data":"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b"} Jul 09 16:28:51.730774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:51.730724 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-d9f7w" podStartSLOduration=1.750686433 podStartE2EDuration="5.730709727s" totalImagesPullingTime="3.980023294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:28:51.730022271 +0000 UTC m=+586.254346605" watchObservedRunningTime="2026-07-09 16:28:51.730709727 +0000 UTC m=+586.255034058" Jul 09 16:28:51.732105 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:51.732073 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" podStartSLOduration=3.92405728 podStartE2EDuration="6.732060148s" totalImagesPullingTime="2.808002868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:28:48.717270728 +0000 UTC m=+583.241595059" watchObservedRunningTime="2026-07-09 16:28:51.732060148 +0000 UTC m=+586.256384520" Jul 09 16:28:59.705226 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:28:59.705190 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:29:04.322446 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.322411 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:29:04.322947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.322651 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" podUID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" containerName="limitador" containerID="cri-o://9b53de0cfa8d2eb6bcb6b4a376c877bfbe22c3fd39b85495b5b194f9985b3c72" gracePeriod=30 Jul 09 16:29:04.778152 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.778107 2577 generic.go:356] "Generic (PLEG): container finished" podID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" containerID="9b53de0cfa8d2eb6bcb6b4a376c877bfbe22c3fd39b85495b5b194f9985b3c72" exitCode=0 Jul 09 16:29:04.778327 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.778173 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" event={"ID":"66f25cca-a183-4e0e-bcb9-efc0c899f8f9","Type":"ContainerDied","Data":"9b53de0cfa8d2eb6bcb6b4a376c877bfbe22c3fd39b85495b5b194f9985b3c72"} Jul 09 16:29:04.889377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.889300 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:29:04.983344 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.983306 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file\") pod \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " Jul 09 16:29:04.983541 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.983411 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58qsn\" (UniqueName: \"kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn\") pod \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\" (UID: \"66f25cca-a183-4e0e-bcb9-efc0c899f8f9\") " Jul 09 16:29:04.983686 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.983662 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file" (OuterVolumeSpecName: "config-file") pod "66f25cca-a183-4e0e-bcb9-efc0c899f8f9" (UID: "66f25cca-a183-4e0e-bcb9-efc0c899f8f9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:29:04.985602 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:04.985569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn" (OuterVolumeSpecName: "kube-api-access-58qsn") pod "66f25cca-a183-4e0e-bcb9-efc0c899f8f9" (UID: "66f25cca-a183-4e0e-bcb9-efc0c899f8f9"). InnerVolumeSpecName "kube-api-access-58qsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:29:05.084347 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.084294 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-config-file\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:05.084347 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.084344 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58qsn\" (UniqueName: \"kubernetes.io/projected/66f25cca-a183-4e0e-bcb9-efc0c899f8f9-kube-api-access-58qsn\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:05.783757 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.783720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" event={"ID":"66f25cca-a183-4e0e-bcb9-efc0c899f8f9","Type":"ContainerDied","Data":"19428b1a1207fc4ad2511817a3a7d488a1401cbfcd15417ae27e5ca9125cf120"} Jul 09 16:29:05.783757 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.783755 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwc4w" Jul 09 16:29:05.784294 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.783784 2577 scope.go:117] "RemoveContainer" containerID="9b53de0cfa8d2eb6bcb6b4a376c877bfbe22c3fd39b85495b5b194f9985b3c72" Jul 09 16:29:05.804987 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.804962 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:29:05.810040 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.810018 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwc4w"] Jul 09 16:29:05.914975 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.914947 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:29:05.919549 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.919523 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:29:05.923580 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.923561 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:29:05.927956 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.927940 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:29:05.993026 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:05.992996 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" path="/var/lib/kubelet/pods/66f25cca-a183-4e0e-bcb9-efc0c899f8f9/volumes" Jul 09 16:29:06.264828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.264773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:06.364489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.364403 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:06.703733 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.703661 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9"] Jul 09 16:29:06.704045 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.704030 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" containerName="limitador" Jul 09 16:29:06.704095 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.704047 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" containerName="limitador" Jul 09 16:29:06.704143 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.704124 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f25cca-a183-4e0e-bcb9-efc0c899f8f9" containerName="limitador" Jul 09 16:29:06.708534 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.708512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.712130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.712107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 16:29:06.712269 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.712109 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tqv6q\"" Jul 09 16:29:06.713165 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.713148 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 16:29:06.713266 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.713191 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 16:29:06.718049 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.718021 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9"] Jul 09 16:29:06.739652 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.739619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:06.797974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.797911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8052c29b-8b80-4a88-afe0-17fed5654f4a-data\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.798335 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.798019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdlb\" (UniqueName: \"kubernetes.io/projected/8052c29b-8b80-4a88-afe0-17fed5654f4a-kube-api-access-gjdlb\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.898772 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.898731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8052c29b-8b80-4a88-afe0-17fed5654f4a-data\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.898981 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.898799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdlb\" (UniqueName: \"kubernetes.io/projected/8052c29b-8b80-4a88-afe0-17fed5654f4a-kube-api-access-gjdlb\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.899219 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.899199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8052c29b-8b80-4a88-afe0-17fed5654f4a-data\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:06.911120 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:06.911097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdlb\" (UniqueName: \"kubernetes.io/projected/8052c29b-8b80-4a88-afe0-17fed5654f4a-kube-api-access-gjdlb\") pod \"postgres-69fb4594cf-m9mq9\" (UID: \"8052c29b-8b80-4a88-afe0-17fed5654f4a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:07.019371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:07.019334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:07.161751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:07.161718 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9"] Jul 09 16:29:07.164853 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:29:07.164824 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8052c29b_8b80_4a88_afe0_17fed5654f4a.slice/crio-98794e941b6c7952e3a92182f431cb1606edd7f323cbf04ccb3d4140108603cd WatchSource:0}: Error finding container 98794e941b6c7952e3a92182f431cb1606edd7f323cbf04ccb3d4140108603cd: Status 404 returned error can't find the container with id 98794e941b6c7952e3a92182f431cb1606edd7f323cbf04ccb3d4140108603cd Jul 09 16:29:07.795265 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:07.795229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" event={"ID":"8052c29b-8b80-4a88-afe0-17fed5654f4a","Type":"ContainerStarted","Data":"98794e941b6c7952e3a92182f431cb1606edd7f323cbf04ccb3d4140108603cd"} Jul 09 16:29:12.823560 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:12.823527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" event={"ID":"8052c29b-8b80-4a88-afe0-17fed5654f4a","Type":"ContainerStarted","Data":"989bf37eedd8d1bcc6d840da922af6f99e808826c8bedba7408db8e862e9e06f"} Jul 09 16:29:12.823974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:12.823632 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:12.841164 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:12.841116 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" podStartSLOduration=1.625718045 podStartE2EDuration="6.84110019s" totalImagesPullingTime="5.215382145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:29:12.839612606 +0000 UTC m=+607.363936938" watchObservedRunningTime="2026-07-09 16:29:12.84110019 +0000 UTC m=+607.365424522" Jul 09 16:29:18.858842 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:18.858808 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-m9mq9" Jul 09 16:29:19.171868 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.171755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:19.356528 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.356478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:19.658115 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.658081 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54df4ccb95-w8kh2"] Jul 09 16:29:19.665582 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.665555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.669297 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.669270 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54df4ccb95-w8kh2"] Jul 09 16:29:19.706279 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.706244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b5r\" (UniqueName: \"kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r\") pod \"authorino-54df4ccb95-w8kh2\" (UID: \"b0c6c701-8f55-49d2-a38f-644ded33784a\") " pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.744064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.744028 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df4ccb95-w8kh2"] Jul 09 16:29:19.744297 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:29:19.744277 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-28b5r], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-54df4ccb95-w8kh2" podUID="b0c6c701-8f55-49d2-a38f-644ded33784a" Jul 09 16:29:19.807690 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.807653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b5r\" (UniqueName: \"kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r\") pod \"authorino-54df4ccb95-w8kh2\" (UID: \"b0c6c701-8f55-49d2-a38f-644ded33784a\") " pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.817275 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.817251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b5r\" (UniqueName: \"kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r\") pod \"authorino-54df4ccb95-w8kh2\" (UID: \"b0c6c701-8f55-49d2-a38f-644ded33784a\") " pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.857958 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.857912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.865533 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.865512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:19.908190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.908112 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28b5r\" (UniqueName: \"kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r\") pod \"b0c6c701-8f55-49d2-a38f-644ded33784a\" (UID: \"b0c6c701-8f55-49d2-a38f-644ded33784a\") " Jul 09 16:29:19.910351 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:19.910320 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r" (OuterVolumeSpecName: "kube-api-access-28b5r") pod "b0c6c701-8f55-49d2-a38f-644ded33784a" (UID: "b0c6c701-8f55-49d2-a38f-644ded33784a"). InnerVolumeSpecName "kube-api-access-28b5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:29:20.009540 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:20.009508 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28b5r\" (UniqueName: \"kubernetes.io/projected/b0c6c701-8f55-49d2-a38f-644ded33784a-kube-api-access-28b5r\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:20.862804 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:20.862774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54df4ccb95-w8kh2" Jul 09 16:29:20.891169 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:20.891135 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54df4ccb95-w8kh2"] Jul 09 16:29:20.893322 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:20.893292 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54df4ccb95-w8kh2"] Jul 09 16:29:21.445586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.445545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:29:21.445766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.445729 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-d9f7w" podUID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" containerName="authorino" containerID="cri-o://3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b" gracePeriod=30 Jul 09 16:29:21.692696 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.692671 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:29:21.725904 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.725815 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brs45\" (UniqueName: \"kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45\") pod \"bedb7f61-a918-44c7-bbe0-ee767b06f80e\" (UID: \"bedb7f61-a918-44c7-bbe0-ee767b06f80e\") " Jul 09 16:29:21.728011 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.727981 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45" (OuterVolumeSpecName: "kube-api-access-brs45") pod "bedb7f61-a918-44c7-bbe0-ee767b06f80e" (UID: "bedb7f61-a918-44c7-bbe0-ee767b06f80e"). InnerVolumeSpecName "kube-api-access-brs45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:29:21.826726 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.826693 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brs45\" (UniqueName: \"kubernetes.io/projected/bedb7f61-a918-44c7-bbe0-ee767b06f80e-kube-api-access-brs45\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:21.868206 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.868166 2577 generic.go:356] "Generic (PLEG): container finished" podID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" containerID="3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b" exitCode=0 Jul 09 16:29:21.868206 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.868202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d9f7w" Jul 09 16:29:21.868420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.868250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d9f7w" event={"ID":"bedb7f61-a918-44c7-bbe0-ee767b06f80e","Type":"ContainerDied","Data":"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b"} Jul 09 16:29:21.868420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.868281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d9f7w" event={"ID":"bedb7f61-a918-44c7-bbe0-ee767b06f80e","Type":"ContainerDied","Data":"556989e6d9b0d5eacf2c49dd5fc826bf8e2edde0c9b82df3446041d8a3140287"} Jul 09 16:29:21.868420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.868298 2577 scope.go:117] "RemoveContainer" containerID="3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b" Jul 09 16:29:21.877038 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.877021 2577 scope.go:117] "RemoveContainer" containerID="3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b" Jul 09 16:29:21.877277 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:29:21.877258 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b\": container with ID starting with 3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b not found: ID does not exist" containerID="3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b" Jul 09 16:29:21.877329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.877287 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b"} err="failed to get container status \"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b\": rpc error: code = NotFound desc = could not find container \"3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b\": container with ID starting with 3307a942a89b06b8772124516c4d70d73e917c230bddad54825cbcad2b0ccc2b not found: ID does not exist" Jul 09 16:29:21.889993 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.889965 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:29:21.893372 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.893339 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d9f7w"] Jul 09 16:29:21.998770 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.998741 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c6c701-8f55-49d2-a38f-644ded33784a" path="/var/lib/kubelet/pods/b0c6c701-8f55-49d2-a38f-644ded33784a/volumes" Jul 09 16:29:21.999002 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:21.998989 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" path="/var/lib/kubelet/pods/bedb7f61-a918-44c7-bbe0-ee767b06f80e/volumes" Jul 09 16:29:25.832590 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.832551 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:25.850248 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.850217 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:25.850573 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.850562 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" containerName="authorino" Jul 09 16:29:25.850622 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.850576 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" containerName="authorino" Jul 09 16:29:25.850655 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.850642 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bedb7f61-a918-44c7-bbe0-ee767b06f80e" containerName="authorino" Jul 09 16:29:25.856274 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.856246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:25.859336 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.859308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 16:29:25.859490 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.859319 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 16:29:25.860689 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.860670 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8zpvg\"" Jul 09 16:29:25.874598 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.874573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:25.960048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.960008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:25.960225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.960061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:25.960225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:25.960082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rt9m\" (UniqueName: \"kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.011853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.011815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:26.047326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.047293 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:29:26.050334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.050317 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.061124 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.061098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.061246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.061173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.061246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.061217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rt9m\" (UniqueName: \"kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.061361 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:29:26.061342 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 16:29:26.061429 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:29:26.061411 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert podName:19412ea0-f702-4388-9068-eb22a8cda1ab nodeName:}" failed. No retries permitted until 2026-07-09 16:29:26.56139092 +0000 UTC m=+621.085715234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert") pod "maas-controller-7ff65dd9d4-twcsh" (UID: "19412ea0-f702-4388-9068-eb22a8cda1ab") : secret "maas-controller-webhook-cert" not found Jul 09 16:29:26.061880 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.061861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.064404 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.064378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:29:26.076326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.076296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rt9m\" (UniqueName: \"kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.107258 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.107124 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:26.161878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.161840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.161878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.161881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.162100 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.161899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcx6\" (UniqueName: \"kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.179173 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.179138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:26.262436 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.262399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.262597 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.262446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.262646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.262615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcx6\" (UniqueName: \"kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.263145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.263123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.265145 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.265120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.271903 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.271881 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcx6\" (UniqueName: \"kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6\") pod \"maas-controller-6bc6967554-qp9lj\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.360505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.360416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:26.501679 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.501651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:29:26.503761 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:29:26.503736 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6900f2a0_b720_427e_b069_336d37578a1c.slice/crio-4e8fe8ce00c8c49875d82e93eaa8323869d6eabd18265e1de9afb2c2cb305f44 WatchSource:0}: Error finding container 4e8fe8ce00c8c49875d82e93eaa8323869d6eabd18265e1de9afb2c2cb305f44: Status 404 returned error can't find the container with id 4e8fe8ce00c8c49875d82e93eaa8323869d6eabd18265e1de9afb2c2cb305f44 Jul 09 16:29:26.565881 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.565846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.568427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.568401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") pod \"maas-controller-7ff65dd9d4-twcsh\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.767142 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.767091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:26.895155 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.895123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-qp9lj" event={"ID":"6900f2a0-b720-427e-b069-336d37578a1c","Type":"ContainerStarted","Data":"4e8fe8ce00c8c49875d82e93eaa8323869d6eabd18265e1de9afb2c2cb305f44"} Jul 09 16:29:26.905545 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:26.905516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:26.908099 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:29:26.908073 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19412ea0_f702_4388_9068_eb22a8cda1ab.slice/crio-1e7181cb3cf7a59b61182216b406ab4ccada21570e749798fe1a3207b0d39468 WatchSource:0}: Error finding container 1e7181cb3cf7a59b61182216b406ab4ccada21570e749798fe1a3207b0d39468: Status 404 returned error can't find the container with id 1e7181cb3cf7a59b61182216b406ab4ccada21570e749798fe1a3207b0d39468 Jul 09 16:29:27.903883 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:27.903739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" event={"ID":"19412ea0-f702-4388-9068-eb22a8cda1ab","Type":"ContainerStarted","Data":"1e7181cb3cf7a59b61182216b406ab4ccada21570e749798fe1a3207b0d39468"} Jul 09 16:29:29.457167 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.457075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:29.584801 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.584759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:29.919221 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.919179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" event={"ID":"19412ea0-f702-4388-9068-eb22a8cda1ab","Type":"ContainerStarted","Data":"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b"} Jul 09 16:29:29.919415 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.919364 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:29.921190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.921152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-qp9lj" event={"ID":"6900f2a0-b720-427e-b069-336d37578a1c","Type":"ContainerStarted","Data":"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f"} Jul 09 16:29:29.921321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.921282 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:29.938351 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.938292 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" podStartSLOduration=2.717853719 podStartE2EDuration="4.938277359s" totalImagesPullingTime="2.22042364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:29:29.937521498 +0000 UTC m=+624.461845834" watchObservedRunningTime="2026-07-09 16:29:29.938277359 +0000 UTC m=+624.462601752" Jul 09 16:29:29.960619 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:29.960572 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bc6967554-qp9lj" podStartSLOduration=1.346992581 podStartE2EDuration="3.960556013s" totalImagesPullingTime="2.613563432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:29:29.958041628 +0000 UTC m=+624.482365960" watchObservedRunningTime="2026-07-09 16:29:29.960556013 +0000 UTC m=+624.484880345" Jul 09 16:29:34.966630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:34.966589 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:35.068117 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:35.068081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:39.384641 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:39.384592 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:39.487191 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:39.487154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:29:40.931947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:40.931901 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:40.932323 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:40.932087 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:29:40.993417 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:40.993385 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:40.993622 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:40.993589 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" podUID="19412ea0-f702-4388-9068-eb22a8cda1ab" containerName="manager" containerID="cri-o://af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b" gracePeriod=30 Jul 09 16:29:41.250933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.250886 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:41.400175 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.400137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") pod \"19412ea0-f702-4388-9068-eb22a8cda1ab\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " Jul 09 16:29:41.400352 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.400198 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca\") pod \"19412ea0-f702-4388-9068-eb22a8cda1ab\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " Jul 09 16:29:41.400352 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.400237 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rt9m\" (UniqueName: \"kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m\") pod \"19412ea0-f702-4388-9068-eb22a8cda1ab\" (UID: \"19412ea0-f702-4388-9068-eb22a8cda1ab\") " Jul 09 16:29:41.400578 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.400554 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "19412ea0-f702-4388-9068-eb22a8cda1ab" (UID: "19412ea0-f702-4388-9068-eb22a8cda1ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:29:41.402633 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.402604 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m" (OuterVolumeSpecName: "kube-api-access-9rt9m") pod "19412ea0-f702-4388-9068-eb22a8cda1ab" (UID: "19412ea0-f702-4388-9068-eb22a8cda1ab"). InnerVolumeSpecName "kube-api-access-9rt9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:29:41.402751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.402676 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "19412ea0-f702-4388-9068-eb22a8cda1ab" (UID: "19412ea0-f702-4388-9068-eb22a8cda1ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:29:41.501044 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.501007 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19412ea0-f702-4388-9068-eb22a8cda1ab-webhook-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:41.501044 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.501036 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19412ea0-f702-4388-9068-eb22a8cda1ab-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:41.501044 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.501049 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rt9m\" (UniqueName: \"kubernetes.io/projected/19412ea0-f702-4388-9068-eb22a8cda1ab-kube-api-access-9rt9m\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:29:41.980891 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.980853 2577 generic.go:356] "Generic (PLEG): container finished" podID="19412ea0-f702-4388-9068-eb22a8cda1ab" containerID="af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b" exitCode=0 Jul 09 16:29:41.981384 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.980909 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" Jul 09 16:29:41.981384 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.980950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" event={"ID":"19412ea0-f702-4388-9068-eb22a8cda1ab","Type":"ContainerDied","Data":"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b"} Jul 09 16:29:41.981384 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.981011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff65dd9d4-twcsh" event={"ID":"19412ea0-f702-4388-9068-eb22a8cda1ab","Type":"ContainerDied","Data":"1e7181cb3cf7a59b61182216b406ab4ccada21570e749798fe1a3207b0d39468"} Jul 09 16:29:41.981384 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.981035 2577 scope.go:117] "RemoveContainer" containerID="af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b" Jul 09 16:29:41.990264 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.990247 2577 scope.go:117] "RemoveContainer" containerID="af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b" Jul 09 16:29:41.990527 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:29:41.990501 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b\": container with ID starting with af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b not found: ID does not exist" containerID="af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b" Jul 09 16:29:41.990604 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:41.990541 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b"} err="failed to get container status \"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b\": rpc error: code = NotFound desc = could not find container \"af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b\": container with ID starting with af0f0b7df190f2ad846d7be64437825d1955a8c4e4dd509d0672a1b02060c67b not found: ID does not exist" Jul 09 16:29:42.002319 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:42.002289 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:42.006888 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:42.006865 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff65dd9d4-twcsh"] Jul 09 16:29:43.993199 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:29:43.993158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19412ea0-f702-4388-9068-eb22a8cda1ab" path="/var/lib/kubelet/pods/19412ea0-f702-4388-9068-eb22a8cda1ab/volumes" Jul 09 16:30:01.940773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:01.940731 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:02.222147 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.222063 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf"] Jul 09 16:30:02.222406 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.222395 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19412ea0-f702-4388-9068-eb22a8cda1ab" containerName="manager" Jul 09 16:30:02.222451 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.222408 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19412ea0-f702-4388-9068-eb22a8cda1ab" containerName="manager" Jul 09 16:30:02.222484 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.222477 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19412ea0-f702-4388-9068-eb22a8cda1ab" containerName="manager" Jul 09 16:30:02.226546 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.226522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.229627 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.229592 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 16:30:02.239941 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.239898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf"] Jul 09 16:30:02.267666 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.267630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.267837 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.267696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea7695-4b83-4f9e-885d-9f4c04e212ed-openshift-service-ca\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.267837 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.267748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9mp\" (UniqueName: \"kubernetes.io/projected/25ea7695-4b83-4f9e-885d-9f4c04e212ed-kube-api-access-rj9mp\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.368633 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.368600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.368846 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.368656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea7695-4b83-4f9e-885d-9f4c04e212ed-openshift-service-ca\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.368846 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.368702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9mp\" (UniqueName: \"kubernetes.io/projected/25ea7695-4b83-4f9e-885d-9f4c04e212ed-kube-api-access-rj9mp\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.368846 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:30:02.368790 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 09 16:30:02.369072 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:30:02.368890 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls podName:25ea7695-4b83-4f9e-885d-9f4c04e212ed nodeName:}" failed. No retries permitted until 2026-07-09 16:30:02.868866478 +0000 UTC m=+657.393190792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls") pod "maas-api-bcf4b654b-8zlrf" (UID: "25ea7695-4b83-4f9e-885d-9f4c04e212ed") : secret "maas-api-serving-cert" not found Jul 09 16:30:02.369432 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.369414 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea7695-4b83-4f9e-885d-9f4c04e212ed-openshift-service-ca\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.379805 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.379776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9mp\" (UniqueName: \"kubernetes.io/projected/25ea7695-4b83-4f9e-885d-9f4c04e212ed-kube-api-access-rj9mp\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.800190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.800146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-68l6r"] Jul 09 16:30:02.803327 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.803302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.806386 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.806358 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 16:30:02.814984 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.814956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-68l6r"] Jul 09 16:30:02.872554 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.872516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.872554 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.872558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtck\" (UniqueName: \"kubernetes.io/projected/172d296a-79ab-4da7-bf3e-f6618d38a54c-kube-api-access-kqtck\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.872766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.872590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172d296a-79ab-4da7-bf3e-f6618d38a54c-plugins-config-volume\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.875215 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.875185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea7695-4b83-4f9e-885d-9f4c04e212ed-maas-api-tls\") pod \"maas-api-bcf4b654b-8zlrf\" (UID: \"25ea7695-4b83-4f9e-885d-9f4c04e212ed\") " pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:02.973333 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.973296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtck\" (UniqueName: \"kubernetes.io/projected/172d296a-79ab-4da7-bf3e-f6618d38a54c-kube-api-access-kqtck\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.973702 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.973341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172d296a-79ab-4da7-bf3e-f6618d38a54c-plugins-config-volume\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.974038 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.974019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/172d296a-79ab-4da7-bf3e-f6618d38a54c-plugins-config-volume\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:02.989756 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:02.989720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtck\" (UniqueName: \"kubernetes.io/projected/172d296a-79ab-4da7-bf3e-f6618d38a54c-kube-api-access-kqtck\") pod \"payload-processing-58f86c7798-68l6r\" (UID: \"172d296a-79ab-4da7-bf3e-f6618d38a54c\") " pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:03.114797 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:03.114700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:03.137634 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:03.137598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:03.272828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:03.272486 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-68l6r"] Jul 09 16:30:03.275989 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:30:03.275958 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172d296a_79ab_4da7_bf3e_f6618d38a54c.slice/crio-0ea200e2168d72a7e45f775ead9e8fffc6dc94102e1f506bbb149665d65ddca0 WatchSource:0}: Error finding container 0ea200e2168d72a7e45f775ead9e8fffc6dc94102e1f506bbb149665d65ddca0: Status 404 returned error can't find the container with id 0ea200e2168d72a7e45f775ead9e8fffc6dc94102e1f506bbb149665d65ddca0 Jul 09 16:30:03.305712 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:03.305688 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf"] Jul 09 16:30:03.307751 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:30:03.307723 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ea7695_4b83_4f9e_885d_9f4c04e212ed.slice/crio-b40dfc3b829de5dac94b81a7d8124aadeda854bbc787509dc2bbdbdaf15d2240 WatchSource:0}: Error finding container b40dfc3b829de5dac94b81a7d8124aadeda854bbc787509dc2bbdbdaf15d2240: Status 404 returned error can't find the container with id b40dfc3b829de5dac94b81a7d8124aadeda854bbc787509dc2bbdbdaf15d2240 Jul 09 16:30:04.083454 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:04.083414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" event={"ID":"25ea7695-4b83-4f9e-885d-9f4c04e212ed","Type":"ContainerStarted","Data":"b40dfc3b829de5dac94b81a7d8124aadeda854bbc787509dc2bbdbdaf15d2240"} Jul 09 16:30:04.084805 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:04.084775 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" event={"ID":"172d296a-79ab-4da7-bf3e-f6618d38a54c","Type":"ContainerStarted","Data":"0ea200e2168d72a7e45f775ead9e8fffc6dc94102e1f506bbb149665d65ddca0"} Jul 09 16:30:04.417018 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:04.416903 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:06.097693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.097660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" event={"ID":"25ea7695-4b83-4f9e-885d-9f4c04e212ed","Type":"ContainerStarted","Data":"81d6fd0e3194215d8f3196ba35901024a69f43911fd052c4158a2dc9e734696f"} Jul 09 16:30:06.098151 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.097766 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:06.099464 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.099436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" event={"ID":"172d296a-79ab-4da7-bf3e-f6618d38a54c","Type":"ContainerStarted","Data":"8b689d60cbb77631c741a01820af1d8f19a38d11d8c21bf93f287af3d6bd76b2"} Jul 09 16:30:06.099596 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.099534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:06.117854 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.117810 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" podStartSLOduration=1.930918289 podStartE2EDuration="4.117793577s" totalImagesPullingTime="2.186875288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:06.116005573 +0000 UTC m=+660.640329909" watchObservedRunningTime="2026-07-09 16:30:06.117793577 +0000 UTC m=+660.642117913" Jul 09 16:30:06.132434 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:06.132386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" podStartSLOduration=1.912584903 podStartE2EDuration="4.132371223s" totalImagesPullingTime="2.21978632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:06.130986255 +0000 UTC m=+660.655310589" watchObservedRunningTime="2026-07-09 16:30:06.132371223 +0000 UTC m=+660.656695554" Jul 09 16:30:12.110213 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:12.110185 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bcf4b654b-8zlrf" Jul 09 16:30:13.063193 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:13.063158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:13.391868 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:13.391762 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:17.107262 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:17.107230 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-68l6r" Jul 09 16:30:24.339652 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.339621 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6"] Jul 09 16:30:24.345340 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.345316 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.348482 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.348460 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 16:30:24.349771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.349742 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2dlbw\"" Jul 09 16:30:24.349881 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.349742 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 16:30:24.350098 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.350035 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 16:30:24.359828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.357532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6"] Jul 09 16:30:24.371473 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.371445 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:24.466513 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.466672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.466672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.466672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.466672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d86b\" (UniqueName: \"kubernetes.io/projected/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-kube-api-access-4d86b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.466672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.466657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.567775 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d86b\" (UniqueName: \"kubernetes.io/projected/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-kube-api-access-4d86b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.567974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568248 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.568227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568288 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.568264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.568584 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.568558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.570534 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.570509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.570699 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.570679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.576603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.576582 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d86b\" (UniqueName: \"kubernetes.io/projected/a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1-kube-api-access-4d86b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6\" (UID: \"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.662401 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.662306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:24.802473 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:24.802444 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6"] Jul 09 16:30:24.804589 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:30:24.804562 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28d7bf5_9c25_4e5b_bb90_f05a0668cfc1.slice/crio-e2d9444d286b54bc09f7978b8c85b1289cbd9770bc775c5ed37e523aaaa00d87 WatchSource:0}: Error finding container e2d9444d286b54bc09f7978b8c85b1289cbd9770bc775c5ed37e523aaaa00d87: Status 404 returned error can't find the container with id e2d9444d286b54bc09f7978b8c85b1289cbd9770bc775c5ed37e523aaaa00d87 Jul 09 16:30:25.196393 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:25.196338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" event={"ID":"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1","Type":"ContainerStarted","Data":"e2d9444d286b54bc09f7978b8c85b1289cbd9770bc775c5ed37e523aaaa00d87"} Jul 09 16:30:26.202700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:26.202665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" event={"ID":"a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1","Type":"ContainerStarted","Data":"fb1330f7e70f260fa3de5e50775b27d6c76dd025c99f75a2d18c473bcf61c82f"} Jul 09 16:30:26.222292 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:26.222240 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" podStartSLOduration=1.039549435 podStartE2EDuration="2.222224313s" totalImagesPullingTime="1.182674878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:26.220228136 +0000 UTC m=+680.744552471" watchObservedRunningTime="2026-07-09 16:30:26.222224313 +0000 UTC m=+680.746548645" Jul 09 16:30:34.662615 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:34.662571 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:34.663140 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:34.662638 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:34.675773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:34.675745 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:35.259413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:35.259386 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6" Jul 09 16:30:39.073847 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:39.073809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:49.130677 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.130643 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd"] Jul 09 16:30:49.135763 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.135743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.139004 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.138979 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 16:30:49.143767 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.143745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd"] Jul 09 16:30:49.179586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.179586 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeafa3b1-667e-457e-a92c-72c900810ef7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.179912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.179912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gmq\" (UniqueName: \"kubernetes.io/projected/eeafa3b1-667e-457e-a92c-72c900810ef7-kube-api-access-x4gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.179912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.179912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.179812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.280845 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.280806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.280861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gmq\" (UniqueName: \"kubernetes.io/projected/eeafa3b1-667e-457e-a92c-72c900810ef7-kube-api-access-x4gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.280886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.280967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.281014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.281029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeafa3b1-667e-457e-a92c-72c900810ef7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281355 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.281332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281419 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.281397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.281478 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.281457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.283430 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.283409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeafa3b1-667e-457e-a92c-72c900810ef7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.283593 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.283576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeafa3b1-667e-457e-a92c-72c900810ef7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.290634 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.290612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gmq\" (UniqueName: \"kubernetes.io/projected/eeafa3b1-667e-457e-a92c-72c900810ef7-kube-api-access-x4gmq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8nxwd\" (UID: \"eeafa3b1-667e-457e-a92c-72c900810ef7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.303125 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.303099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:30:49.446971 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.446862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:49.588912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:49.588832 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd"] Jul 09 16:30:49.591803 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:30:49.591777 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeafa3b1_667e_457e_a92c_72c900810ef7.slice/crio-1985a54ef057c48b997ec0be07c532f577daafb7cca87fe5bf63b407bb93726b WatchSource:0}: Error finding container 1985a54ef057c48b997ec0be07c532f577daafb7cca87fe5bf63b407bb93726b: Status 404 returned error can't find the container with id 1985a54ef057c48b997ec0be07c532f577daafb7cca87fe5bf63b407bb93726b Jul 09 16:30:50.318890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:50.318853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" event={"ID":"eeafa3b1-667e-457e-a92c-72c900810ef7","Type":"ContainerStarted","Data":"61140becc10e9f35318286eec7427e05d1f0b2b84ac17acf1aea2eeab17c17da"} Jul 09 16:30:50.319297 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:50.318906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" event={"ID":"eeafa3b1-667e-457e-a92c-72c900810ef7","Type":"ContainerStarted","Data":"1985a54ef057c48b997ec0be07c532f577daafb7cca87fe5bf63b407bb93726b"} Jul 09 16:30:50.339388 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:50.339334 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" podStartSLOduration=1.149509387 podStartE2EDuration="1.339317841s" totalImagesPullingTime="189.808454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:50.337662884 +0000 UTC m=+704.861987237" watchObservedRunningTime="2026-07-09 16:30:50.339317841 +0000 UTC m=+704.863642172" Jul 09 16:30:59.447892 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:59.447798 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:59.447892 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:59.447883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:30:59.460903 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:30:59.460870 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:31:00.377873 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:00.377839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8nxwd" Jul 09 16:31:18.692666 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:18.692624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:31:18.747448 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:18.747403 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:31:20.502399 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:20.502365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:31:26.870966 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:26.870903 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:31:31.842587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:31.842555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:31:38.504631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:31:38.504586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:32:16.049698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:32:16.049665 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:32:16.155931 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:32:16.155889 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:32:17.171612 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:32:17.171571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:32:17.267744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:32:17.267715 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:33:27.416796 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.416741 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:33:27.417379 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.416980 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bc6967554-qp9lj" podUID="6900f2a0-b720-427e-b069-336d37578a1c" containerName="manager" containerID="cri-o://9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f" gracePeriod=30 Jul 09 16:33:27.673535 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.673474 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:33:27.767048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.767015 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert\") pod \"6900f2a0-b720-427e-b069-336d37578a1c\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " Jul 09 16:33:27.767048 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.767047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca\") pod \"6900f2a0-b720-427e-b069-336d37578a1c\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " Jul 09 16:33:27.767281 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.767081 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqcx6\" (UniqueName: \"kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6\") pod \"6900f2a0-b720-427e-b069-336d37578a1c\" (UID: \"6900f2a0-b720-427e-b069-336d37578a1c\") " Jul 09 16:33:27.767457 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.767430 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "6900f2a0-b720-427e-b069-336d37578a1c" (UID: "6900f2a0-b720-427e-b069-336d37578a1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:33:27.769267 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.769240 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6" (OuterVolumeSpecName: "kube-api-access-hqcx6") pod "6900f2a0-b720-427e-b069-336d37578a1c" (UID: "6900f2a0-b720-427e-b069-336d37578a1c"). InnerVolumeSpecName "kube-api-access-hqcx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:33:27.769441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.769421 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6900f2a0-b720-427e-b069-336d37578a1c" (UID: "6900f2a0-b720-427e-b069-336d37578a1c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:33:27.868057 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.868026 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6900f2a0-b720-427e-b069-336d37578a1c-webhook-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:33:27.868057 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.868054 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6900f2a0-b720-427e-b069-336d37578a1c-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:33:27.868057 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.868063 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcx6\" (UniqueName: \"kubernetes.io/projected/6900f2a0-b720-427e-b069-336d37578a1c-kube-api-access-hqcx6\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:33:27.998050 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:27.998014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:33:28.081271 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.081229 2577 generic.go:356] "Generic (PLEG): container finished" podID="6900f2a0-b720-427e-b069-336d37578a1c" containerID="9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f" exitCode=0 Jul 09 16:33:28.081428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.081278 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-qp9lj" Jul 09 16:33:28.081428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.081311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-qp9lj" event={"ID":"6900f2a0-b720-427e-b069-336d37578a1c","Type":"ContainerDied","Data":"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f"} Jul 09 16:33:28.081428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.081372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-qp9lj" event={"ID":"6900f2a0-b720-427e-b069-336d37578a1c","Type":"ContainerDied","Data":"4e8fe8ce00c8c49875d82e93eaa8323869d6eabd18265e1de9afb2c2cb305f44"} Jul 09 16:33:28.081428 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.081402 2577 scope.go:117] "RemoveContainer" containerID="9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f" Jul 09 16:33:28.090235 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.090218 2577 scope.go:117] "RemoveContainer" containerID="9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f" Jul 09 16:33:28.090530 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:33:28.090505 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f\": container with ID starting with 9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f not found: ID does not exist" containerID="9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f" Jul 09 16:33:28.090599 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.090543 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f"} err="failed to get container status \"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f\": rpc error: code = NotFound desc = could not find container \"9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f\": container with ID starting with 9f83291357d71f95ea0db0b2076f71a7c1a89d0d309c8f6f372bac311168f16f not found: ID does not exist" Jul 09 16:33:28.098785 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.098758 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:33:28.103042 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.103009 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bc6967554-qp9lj"] Jul 09 16:33:28.748093 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.748052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:33:28.861646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.861614 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bc6967554-zdbsq"] Jul 09 16:33:28.862113 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.862097 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6900f2a0-b720-427e-b069-336d37578a1c" containerName="manager" Jul 09 16:33:28.862171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.862114 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6900f2a0-b720-427e-b069-336d37578a1c" containerName="manager" Jul 09 16:33:28.862223 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.862214 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6900f2a0-b720-427e-b069-336d37578a1c" containerName="manager" Jul 09 16:33:28.866469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.866451 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.870614 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.870589 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 16:33:28.870933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.870652 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 16:33:28.870933 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.870595 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8zpvg\"" Jul 09 16:33:28.874511 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.874488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-webhook-cert\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.874698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.874670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxms\" (UniqueName: \"kubernetes.io/projected/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-kube-api-access-vtxms\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.874806 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.874762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-service-ca\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.876262 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.876242 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6967554-zdbsq"] Jul 09 16:33:28.975552 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.975507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-webhook-cert\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.975878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.975559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxms\" (UniqueName: \"kubernetes.io/projected/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-kube-api-access-vtxms\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.975878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.975615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-service-ca\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.976333 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.976314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-service-ca\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.978103 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.978086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-webhook-cert\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:28.986096 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:28.986058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxms\" (UniqueName: \"kubernetes.io/projected/13464a1c-7fbf-47f4-84af-ef7cabc3b2da-kube-api-access-vtxms\") pod \"maas-controller-6bc6967554-zdbsq\" (UID: \"13464a1c-7fbf-47f4-84af-ef7cabc3b2da\") " pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:29.178218 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:29.178124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:29.523334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:29.523311 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6967554-zdbsq"] Jul 09 16:33:29.525574 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:33:29.525541 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13464a1c_7fbf_47f4_84af_ef7cabc3b2da.slice/crio-825d6e8f5e2b327c5b7a943b1ac3cb07a53cd00ee387fb607879e3bc635782c7 WatchSource:0}: Error finding container 825d6e8f5e2b327c5b7a943b1ac3cb07a53cd00ee387fb607879e3bc635782c7: Status 404 returned error can't find the container with id 825d6e8f5e2b327c5b7a943b1ac3cb07a53cd00ee387fb607879e3bc635782c7 Jul 09 16:33:29.527000 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:29.526983 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:33:29.992997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:29.992966 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6900f2a0-b720-427e-b069-336d37578a1c" path="/var/lib/kubelet/pods/6900f2a0-b720-427e-b069-336d37578a1c/volumes" Jul 09 16:33:30.093323 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:30.093287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-zdbsq" event={"ID":"13464a1c-7fbf-47f4-84af-ef7cabc3b2da","Type":"ContainerStarted","Data":"825d6e8f5e2b327c5b7a943b1ac3cb07a53cd00ee387fb607879e3bc635782c7"} Jul 09 16:33:31.099510 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:31.099473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6967554-zdbsq" event={"ID":"13464a1c-7fbf-47f4-84af-ef7cabc3b2da","Type":"ContainerStarted","Data":"bb3b668bf0e3376c974251e805db9ff117a595194ae6f4acb02a75994f48c676"} Jul 09 16:33:31.099852 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:31.099540 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:33:31.116504 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:31.116272 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bc6967554-zdbsq" podStartSLOduration=1.678402149 podStartE2EDuration="3.116253001s" totalImagesPullingTime="1.437850852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:33:31.115725582 +0000 UTC m=+865.640049927" watchObservedRunningTime="2026-07-09 16:33:31.116253001 +0000 UTC m=+865.640577335" Jul 09 16:33:42.111133 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:33:42.111100 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bc6967554-zdbsq" Jul 09 16:34:05.946800 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:05.946762 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:34:05.948625 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:05.948581 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:34:05.955691 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:05.955667 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:34:05.957090 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:05.957065 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:34:10.359488 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:10.359444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:10.459759 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:10.459716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:10.559778 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:10.559747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:16.307718 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:16.307682 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:16.432438 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:16.432397 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:16.558860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:16.558768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:16.669615 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:16.669564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:22.407230 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:22.407181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:34:22.508584 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:34:22.508550 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:36:27.926360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:36:27.926330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:36:55.178325 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:36:55.178289 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:37:02.498998 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:37:02.498963 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:38:15.927224 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:38:15.927187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:39:05.977585 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:39:05.977553 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:39:05.978125 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:39:05.977794 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:39:05.985826 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:39:05.985808 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:39:05.986169 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:39:05.986152 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:41:04.389871 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:41:04.389824 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:44:06.006000 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:44:06.005966 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:44:06.008633 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:44:06.008611 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:44:06.014992 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:44:06.014970 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:44:06.017523 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:44:06.017505 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:45:00.148126 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.148046 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:45:00.151572 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.151552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:45:00.154577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.154552 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2fl4\"" Jul 09 16:45:00.167063 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.167040 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:45:00.287420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.287376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4b6b\" (UniqueName: \"kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b\") pod \"maas-api-key-cleanup-29726925-jgvfc\" (UID: \"bfdf81f1-b12b-4658-a3fa-7fe8160df94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:45:00.388634 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.388597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4b6b\" (UniqueName: \"kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b\") pod \"maas-api-key-cleanup-29726925-jgvfc\" (UID: \"bfdf81f1-b12b-4658-a3fa-7fe8160df94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:45:00.398840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.398765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4b6b\" (UniqueName: \"kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b\") pod \"maas-api-key-cleanup-29726925-jgvfc\" (UID: \"bfdf81f1-b12b-4658-a3fa-7fe8160df94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:45:00.462179 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.462147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:45:00.603588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.603552 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:45:00.606323 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:45:00.606290 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf81f1_b12b_4658_a3fa_7fe8160df94a.slice/crio-86b9eac5bf5c32c30ebd105d70c26435c38d20866e1be8cbf8bb292bd7a92f8c WatchSource:0}: Error finding container 86b9eac5bf5c32c30ebd105d70c26435c38d20866e1be8cbf8bb292bd7a92f8c: Status 404 returned error can't find the container with id 86b9eac5bf5c32c30ebd105d70c26435c38d20866e1be8cbf8bb292bd7a92f8c Jul 09 16:45:00.608250 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:00.608235 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:45:01.434440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:01.434404 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" event={"ID":"bfdf81f1-b12b-4658-a3fa-7fe8160df94a","Type":"ContainerStarted","Data":"86b9eac5bf5c32c30ebd105d70c26435c38d20866e1be8cbf8bb292bd7a92f8c"} Jul 09 16:45:04.452514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:04.452475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" event={"ID":"bfdf81f1-b12b-4658-a3fa-7fe8160df94a","Type":"ContainerStarted","Data":"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15"} Jul 09 16:45:04.470733 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:04.470684 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" podStartSLOduration=1.66032264 podStartE2EDuration="4.470667788s" totalImagesPullingTime="2.810345148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:04.469556165 +0000 UTC m=+1558.993880498" watchObservedRunningTime="2026-07-09 16:45:04.470667788 +0000 UTC m=+1558.994992120" Jul 09 16:45:15.641335 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:15.641296 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:22.452807 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:22.452776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:30.679765 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:30.679726 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:37.172220 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:37.172174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:37.272532 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:37.272502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:38.625877 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:38.625841 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:38.724061 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:38.724022 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:39.056183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:39.056148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:44.964608 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:44.964565 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:45.067661 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:45.067624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:46.450907 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.450861 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:46.457840 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.457819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.460571 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.460545 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a5f599bc-serving-cert\"" Jul 09 16:45:46.465003 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.464974 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:46.599314 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.599275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.599314 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.599316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.599564 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.599400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2q78\" (UniqueName: \"kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.700549 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.700505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.700549 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.700547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.700807 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.700610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2q78\" (UniqueName: \"kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.700807 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:45:46.700664 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-serving-cert: secret "maas-api-e2e-ait-a5f599bc-serving-cert" not found Jul 09 16:45:46.700807 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:45:46.700727 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls podName:580f184f-84cf-4498-a0c2-99af8bb6a8e2 nodeName:}" failed. No retries permitted until 2026-07-09 16:45:47.200709976 +0000 UTC m=+1601.725034287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls") pod "maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" (UID: "580f184f-84cf-4498-a0c2-99af8bb6a8e2") : secret "maas-api-e2e-ait-a5f599bc-serving-cert" not found Jul 09 16:45:46.701341 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.701293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.710201 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.710180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2q78\" (UniqueName: \"kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:46.821087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.821053 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:46.927481 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:46.927440 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:47.021178 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.021148 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:47.122133 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.122093 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:47.206239 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.206200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:47.208791 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.208766 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") pod \"maas-api-e2e-ait-a5f599bc-67886c569-nmwcf\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:47.371673 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.371578 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:47.524786 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.524716 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:47.527463 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:45:47.527434 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580f184f_84cf_4498_a0c2_99af8bb6a8e2.slice/crio-72c5ae2d402203e84e68d197c44209e9e4d2e522182524db0f9621ab2369f67a WatchSource:0}: Error finding container 72c5ae2d402203e84e68d197c44209e9e4d2e522182524db0f9621ab2369f67a: Status 404 returned error can't find the container with id 72c5ae2d402203e84e68d197c44209e9e4d2e522182524db0f9621ab2369f67a Jul 09 16:45:47.670114 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:47.670031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" event={"ID":"580f184f-84cf-4498-a0c2-99af8bb6a8e2","Type":"ContainerStarted","Data":"72c5ae2d402203e84e68d197c44209e9e4d2e522182524db0f9621ab2369f67a"} Jul 09 16:45:48.677176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:48.677079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" event={"ID":"580f184f-84cf-4498-a0c2-99af8bb6a8e2","Type":"ContainerStarted","Data":"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5"} Jul 09 16:45:48.677176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:48.677157 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:48.697527 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:48.697479 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" podStartSLOduration=1.876278803 podStartE2EDuration="2.697464589s" totalImagesPullingTime="821.185786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:48.695431492 +0000 UTC m=+1603.219755834" watchObservedRunningTime="2026-07-09 16:45:48.697464589 +0000 UTC m=+1603.221788920" Jul 09 16:45:49.136878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:49.136830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:54.371563 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.371529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:54.372015 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.371752 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" podUID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" containerName="maas-api" containerID="cri-o://e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5" gracePeriod=30 Jul 09 16:45:54.378970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.378941 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:54.400144 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.400089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:54.456440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.456411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:54.579834 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.579799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:54.634721 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.634657 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:54.666954 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.666905 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:45:54.707399 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.707362 2577 generic.go:356] "Generic (PLEG): container finished" podID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" containerID="e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5" exitCode=0 Jul 09 16:45:54.707587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.707414 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" Jul 09 16:45:54.707587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.707438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" event={"ID":"580f184f-84cf-4498-a0c2-99af8bb6a8e2","Type":"ContainerDied","Data":"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5"} Jul 09 16:45:54.707587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.707490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf" event={"ID":"580f184f-84cf-4498-a0c2-99af8bb6a8e2","Type":"ContainerDied","Data":"72c5ae2d402203e84e68d197c44209e9e4d2e522182524db0f9621ab2369f67a"} Jul 09 16:45:54.707587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.707521 2577 scope.go:117] "RemoveContainer" containerID="e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5" Jul 09 16:45:54.716363 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.716345 2577 scope.go:117] "RemoveContainer" containerID="e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5" Jul 09 16:45:54.716616 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:45:54.716595 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5\": container with ID starting with e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5 not found: ID does not exist" containerID="e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5" Jul 09 16:45:54.716678 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.716629 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5"} err="failed to get container status \"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5\": rpc error: code = NotFound desc = could not find container \"e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5\": container with ID starting with e142c64005c7f78468249317d9c2031d7f4692c74151cc52e69a5c25253f9cf5 not found: ID does not exist" Jul 09 16:45:54.775584 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.775549 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") pod \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " Jul 09 16:45:54.775584 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.775587 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca\") pod \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " Jul 09 16:45:54.775769 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.775634 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2q78\" (UniqueName: \"kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78\") pod \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\" (UID: \"580f184f-84cf-4498-a0c2-99af8bb6a8e2\") " Jul 09 16:45:54.776068 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.776039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "580f184f-84cf-4498-a0c2-99af8bb6a8e2" (UID: "580f184f-84cf-4498-a0c2-99af8bb6a8e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:45:54.778102 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.778074 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78" (OuterVolumeSpecName: "kube-api-access-k2q78") pod "580f184f-84cf-4498-a0c2-99af8bb6a8e2" (UID: "580f184f-84cf-4498-a0c2-99af8bb6a8e2"). InnerVolumeSpecName "kube-api-access-k2q78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:45:54.778102 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.778090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "580f184f-84cf-4498-a0c2-99af8bb6a8e2" (UID: "580f184f-84cf-4498-a0c2-99af8bb6a8e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:45:54.877161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.877119 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/580f184f-84cf-4498-a0c2-99af8bb6a8e2-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:45:54.877161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.877153 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/580f184f-84cf-4498-a0c2-99af8bb6a8e2-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:45:54.877161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:54.877163 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2q78\" (UniqueName: \"kubernetes.io/projected/580f184f-84cf-4498-a0c2-99af8bb6a8e2-kube-api-access-k2q78\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:45:55.029313 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:55.029281 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:55.032772 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:55.032732 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a5f599bc-67886c569-nmwcf"] Jul 09 16:45:55.993637 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:55.993605 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" path="/var/lib/kubelet/pods/580f184f-84cf-4498-a0c2-99af8bb6a8e2/volumes" Jul 09 16:45:56.367398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:45:56.367368 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:00.009510 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.009473 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:46:00.009945 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.009678 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" podUID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" containerName="cleanup" containerID="cri-o://ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15" gracePeriod=30 Jul 09 16:46:00.423119 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.423035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:00.500589 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.500555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:00.646730 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.646684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:00.745046 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:00.745015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:01.814890 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:01.814849 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:01.913769 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:01.913734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:02.050674 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.050642 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:02.051292 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.051272 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" containerName="maas-api" Jul 09 16:46:02.051362 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.051298 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" containerName="maas-api" Jul 09 16:46:02.051418 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.051406 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="580f184f-84cf-4498-a0c2-99af8bb6a8e2" containerName="maas-api" Jul 09 16:46:02.056105 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.056080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.058906 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.058883 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5aa593b6-serving-cert\"" Jul 09 16:46:02.062887 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.062856 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:02.087360 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.087270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:02.138669 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.138626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964lf\" (UniqueName: \"kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.138669 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.138678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.138911 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.138748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.239720 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.239682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964lf\" (UniqueName: \"kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.239934 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.239730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.239934 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.239758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.239934 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:02.239854 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-serving-cert: secret "maas-api-e2e-ait-5aa593b6-serving-cert" not found Jul 09 16:46:02.240125 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:02.239955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls podName:2bc8e7e1-8712-4e22-a46d-dd79135c8667 nodeName:}" failed. No retries permitted until 2026-07-09 16:46:02.739934349 +0000 UTC m=+1617.264258679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls") pod "maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" (UID: "2bc8e7e1-8712-4e22-a46d-dd79135c8667") : secret "maas-api-e2e-ait-5aa593b6-serving-cert" not found Jul 09 16:46:02.240409 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.240392 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.248749 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.248726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-964lf\" (UniqueName: \"kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.368154 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.368055 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:02.690373 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.690271 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:02.744774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.744737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.747837 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.747802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") pod \"maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:02.791034 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.791003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:02.969848 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:02.969763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:03.117816 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:03.117788 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:03.120190 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:46:03.120161 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc8e7e1_8712_4e22_a46d_dd79135c8667.slice/crio-f0b8097d0888dcd22b90a6bd0e93c3d22148948c69bb628658472d118e24f5b6 WatchSource:0}: Error finding container f0b8097d0888dcd22b90a6bd0e93c3d22148948c69bb628658472d118e24f5b6: Status 404 returned error can't find the container with id f0b8097d0888dcd22b90a6bd0e93c3d22148948c69bb628658472d118e24f5b6 Jul 09 16:46:03.755686 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:03.755650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" event={"ID":"2bc8e7e1-8712-4e22-a46d-dd79135c8667","Type":"ContainerStarted","Data":"f0b8097d0888dcd22b90a6bd0e93c3d22148948c69bb628658472d118e24f5b6"} Jul 09 16:46:03.912126 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:03.912096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:05.768523 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:05.768486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" event={"ID":"2bc8e7e1-8712-4e22-a46d-dd79135c8667","Type":"ContainerStarted","Data":"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92"} Jul 09 16:46:05.768969 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:05.768690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:05.786732 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:05.786536 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" podStartSLOduration=1.984714152 podStartE2EDuration="3.786516802s" totalImagesPullingTime="1.80180265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:46:05.785815625 +0000 UTC m=+1620.310139958" watchObservedRunningTime="2026-07-09 16:46:05.786516802 +0000 UTC m=+1620.310841135" Jul 09 16:46:09.790114 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:09.790080 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:09.790622 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:09.790373 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" podUID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" containerName="maas-api" containerID="cri-o://b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92" gracePeriod=30 Jul 09 16:46:09.797110 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:09.797084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:09.819022 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:09.818991 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:09.888857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:09.888782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:10.047434 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.047378 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:10.113707 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.113677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca\") pod \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " Jul 09 16:46:10.113884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.113743 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-964lf\" (UniqueName: \"kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf\") pod \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " Jul 09 16:46:10.113884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.113766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") pod \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\" (UID: \"2bc8e7e1-8712-4e22-a46d-dd79135c8667\") " Jul 09 16:46:10.114181 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.114142 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bc8e7e1-8712-4e22-a46d-dd79135c8667" (UID: "2bc8e7e1-8712-4e22-a46d-dd79135c8667"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:46:10.116045 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.116016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf" (OuterVolumeSpecName: "kube-api-access-964lf") pod "2bc8e7e1-8712-4e22-a46d-dd79135c8667" (UID: "2bc8e7e1-8712-4e22-a46d-dd79135c8667"). InnerVolumeSpecName "kube-api-access-964lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:10.116134 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.116051 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bc8e7e1-8712-4e22-a46d-dd79135c8667" (UID: "2bc8e7e1-8712-4e22-a46d-dd79135c8667"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:46:10.214970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.214906 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8e7e1-8712-4e22-a46d-dd79135c8667-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:10.214970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.214970 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964lf\" (UniqueName: \"kubernetes.io/projected/2bc8e7e1-8712-4e22-a46d-dd79135c8667-kube-api-access-964lf\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:10.215176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.214983 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc8e7e1-8712-4e22-a46d-dd79135c8667-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:10.769354 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.769308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:10.793874 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.793833 2577 generic.go:356] "Generic (PLEG): container finished" podID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" containerID="b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92" exitCode=0 Jul 09 16:46:10.794307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.793902 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" Jul 09 16:46:10.794307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.793951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" event={"ID":"2bc8e7e1-8712-4e22-a46d-dd79135c8667","Type":"ContainerDied","Data":"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92"} Jul 09 16:46:10.794307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.794027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2" event={"ID":"2bc8e7e1-8712-4e22-a46d-dd79135c8667","Type":"ContainerDied","Data":"f0b8097d0888dcd22b90a6bd0e93c3d22148948c69bb628658472d118e24f5b6"} Jul 09 16:46:10.794307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.794050 2577 scope.go:117] "RemoveContainer" containerID="b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92" Jul 09 16:46:10.803700 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.803682 2577 scope.go:117] "RemoveContainer" containerID="b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92" Jul 09 16:46:10.803997 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:10.803976 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92\": container with ID starting with b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92 not found: ID does not exist" containerID="b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92" Jul 09 16:46:10.804075 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.804006 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92"} err="failed to get container status \"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92\": rpc error: code = NotFound desc = could not find container \"b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92\": container with ID starting with b73d5345e7d60bb7d4dfde3480c7e44906ac0797ffe037b48e58b3d266343d92 not found: ID does not exist" Jul 09 16:46:10.817148 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.817121 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:10.825130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.825105 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aa593b6-59b56d84c5-72kp2"] Jul 09 16:46:10.843195 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.843162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:10.918387 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:10.918357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:11.993307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:11.993275 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" path="/var/lib/kubelet/pods/2bc8e7e1-8712-4e22-a46d-dd79135c8667/volumes" Jul 09 16:46:15.954306 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:15.954270 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:16.733569 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:16.733529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:16.834822 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:16.834779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:17.088232 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:17.088143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:17.188123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:17.188078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.162431 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.162389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.236034 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.236004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:18.236602 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.236585 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" containerName="maas-api" Jul 09 16:46:18.236663 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.236606 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" containerName="maas-api" Jul 09 16:46:18.236716 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.236704 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc8e7e1-8712-4e22-a46d-dd79135c8667" containerName="maas-api" Jul 09 16:46:18.241646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.241612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.244362 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.244342 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5e823c1a-serving-cert\"" Jul 09 16:46:18.254730 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.254707 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:18.262559 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.262527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.286331 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.286295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kc7t\" (UniqueName: \"kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.286495 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.286350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.286495 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.286417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.338107 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.338063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.387722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.387685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.387887 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.387763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kc7t\" (UniqueName: \"kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.387887 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.387793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.388010 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:18.387904 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-serving-cert: secret "maas-api-e2e-derive-5e823c1a-serving-cert" not found Jul 09 16:46:18.388010 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:18.387990 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls podName:72c0e142-810e-4a8b-a5ac-36d1d2e6b34b nodeName:}" failed. No retries permitted until 2026-07-09 16:46:18.887970109 +0000 UTC m=+1633.412294439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls") pod "maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" (UID: "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b") : secret "maas-api-e2e-derive-5e823c1a-serving-cert" not found Jul 09 16:46:18.388343 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.388322 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.404769 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.404734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kc7t\" (UniqueName: \"kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.597587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.597542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.863460 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.863381 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:18.893152 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.893116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.895692 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.895666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") pod \"maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:18.964731 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:18.964690 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:19.152377 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.152292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:19.299439 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.299414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:19.301536 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:46:19.301510 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c0e142_810e_4a8b_a5ac_36d1d2e6b34b.slice/crio-1f810427d50bb1699a0ff55a740ab616651f5d5ec022f2c3ebf1513e8595023f WatchSource:0}: Error finding container 1f810427d50bb1699a0ff55a740ab616651f5d5ec022f2c3ebf1513e8595023f: Status 404 returned error can't find the container with id 1f810427d50bb1699a0ff55a740ab616651f5d5ec022f2c3ebf1513e8595023f Jul 09 16:46:19.846882 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.846835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" event={"ID":"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b","Type":"ContainerStarted","Data":"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88"} Jul 09 16:46:19.847087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.846908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" event={"ID":"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b","Type":"ContainerStarted","Data":"1f810427d50bb1699a0ff55a740ab616651f5d5ec022f2c3ebf1513e8595023f"} Jul 09 16:46:19.847087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.846958 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:19.866522 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:19.866428 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" podStartSLOduration=1.5609064 podStartE2EDuration="1.866411137s" totalImagesPullingTime="305.504737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:46:19.865359553 +0000 UTC m=+1634.389683886" watchObservedRunningTime="2026-07-09 16:46:19.866411137 +0000 UTC m=+1634.390735468" Jul 09 16:46:20.667317 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:20.667281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:22.473459 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:22.473422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:22.573133 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:22.573100 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:22.691041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:22.691006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:25.488780 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.488748 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:25.489223 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.489014 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" podUID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" containerName="maas-api" containerID="cri-o://ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88" gracePeriod=30 Jul 09 16:46:25.495255 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.495233 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:25.511610 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.511583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:25.611104 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.611073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:25.683387 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.683342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:25.738409 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.738386 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:25.823630 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.823543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:25.859857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.859824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca\") pod \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " Jul 09 16:46:25.860053 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.859930 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") pod \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " Jul 09 16:46:25.860053 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.859987 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kc7t\" (UniqueName: \"kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t\") pod \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\" (UID: \"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b\") " Jul 09 16:46:25.860243 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.860213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" (UID: "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:46:25.862063 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.862032 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" (UID: "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:46:25.862166 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.862110 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t" (OuterVolumeSpecName: "kube-api-access-7kc7t") pod "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" (UID: "72c0e142-810e-4a8b-a5ac-36d1d2e6b34b"). InnerVolumeSpecName "kube-api-access-7kc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:25.878267 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.878227 2577 generic.go:356] "Generic (PLEG): container finished" podID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" containerID="ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88" exitCode=0 Jul 09 16:46:25.878267 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.878263 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" Jul 09 16:46:25.878441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.878288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" event={"ID":"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b","Type":"ContainerDied","Data":"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88"} Jul 09 16:46:25.878441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.878345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc" event={"ID":"72c0e142-810e-4a8b-a5ac-36d1d2e6b34b","Type":"ContainerDied","Data":"1f810427d50bb1699a0ff55a740ab616651f5d5ec022f2c3ebf1513e8595023f"} Jul 09 16:46:25.878441 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.878367 2577 scope.go:117] "RemoveContainer" containerID="ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88" Jul 09 16:46:25.887500 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.887483 2577 scope.go:117] "RemoveContainer" containerID="ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88" Jul 09 16:46:25.887753 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:25.887733 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88\": container with ID starting with ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88 not found: ID does not exist" containerID="ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88" Jul 09 16:46:25.887815 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.887765 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88"} err="failed to get container status \"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88\": rpc error: code = NotFound desc = could not find container \"ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88\": container with ID starting with ce575fd26e6f597640f8691763683e66af0c2082fdd880b643200d09061d1a88 not found: ID does not exist" Jul 09 16:46:25.914773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.914740 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:25.917438 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.917412 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5e823c1a-cd5d7b8c8-rvlsc"] Jul 09 16:46:25.960717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.960687 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:25.960717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.960714 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kc7t\" (UniqueName: \"kubernetes.io/projected/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-kube-api-access-7kc7t\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:25.960717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.960724 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:25.994242 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:25.994211 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" path="/var/lib/kubelet/pods/72c0e142-810e-4a8b-a5ac-36d1d2e6b34b/volumes" Jul 09 16:46:27.461799 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:27.461761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:30.213523 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.213489 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:46:30.301857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.301759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4b6b\" (UniqueName: \"kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b\") pod \"bfdf81f1-b12b-4658-a3fa-7fe8160df94a\" (UID: \"bfdf81f1-b12b-4658-a3fa-7fe8160df94a\") " Jul 09 16:46:30.304087 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.304050 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b" (OuterVolumeSpecName: "kube-api-access-c4b6b") pod "bfdf81f1-b12b-4658-a3fa-7fe8160df94a" (UID: "bfdf81f1-b12b-4658-a3fa-7fe8160df94a"). InnerVolumeSpecName "kube-api-access-c4b6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:30.402774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.402737 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4b6b\" (UniqueName: \"kubernetes.io/projected/bfdf81f1-b12b-4658-a3fa-7fe8160df94a-kube-api-access-c4b6b\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:46:30.908356 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.908317 2577 generic.go:356] "Generic (PLEG): container finished" podID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" containerID="ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15" exitCode=137 Jul 09 16:46:30.908587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.908364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" event={"ID":"bfdf81f1-b12b-4658-a3fa-7fe8160df94a","Type":"ContainerDied","Data":"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15"} Jul 09 16:46:30.908587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.908378 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" Jul 09 16:46:30.908587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.908406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc" event={"ID":"bfdf81f1-b12b-4658-a3fa-7fe8160df94a","Type":"ContainerDied","Data":"86b9eac5bf5c32c30ebd105d70c26435c38d20866e1be8cbf8bb292bd7a92f8c"} Jul 09 16:46:30.908587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.908428 2577 scope.go:117] "RemoveContainer" containerID="ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15" Jul 09 16:46:30.917459 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.917442 2577 scope.go:117] "RemoveContainer" containerID="ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15" Jul 09 16:46:30.917732 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:46:30.917714 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15\": container with ID starting with ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15 not found: ID does not exist" containerID="ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15" Jul 09 16:46:30.917773 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.917741 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15"} err="failed to get container status \"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15\": rpc error: code = NotFound desc = could not find container \"ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15\": container with ID starting with ca3507bf97530d73e67fc1f436c7d9c1028a8bb42ce3f9ed3f86b4970f04ff15 not found: ID does not exist" Jul 09 16:46:30.930019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.929991 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:46:30.933729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:30.933704 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-jgvfc"] Jul 09 16:46:31.891155 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:31.891118 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:31.992768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:31.992728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:31.993768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:31.993742 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" path="/var/lib/kubelet/pods/bfdf81f1-b12b-4658-a3fa-7fe8160df94a/volumes" Jul 09 16:46:32.365425 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:32.365390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:32.465352 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:32.465320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:32.823234 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:32.823198 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:46:38.220806 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:46:38.220764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:03.840268 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:03.840228 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:09.732886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:09.732845 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:15.456948 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:15.456885 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:15.519624 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:15.519585 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:15.609974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:15.609907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:15.712497 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:15.712410 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:15.821585 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:15.821548 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:16.255534 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:16.255500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:42.248392 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:42.248308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:47.940125 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:47.940091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:48.040997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:48.040964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:47:48.140271 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:47:48.140242 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:04.025289 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:04.025256 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:09.990376 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:09.990333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:10.086898 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:10.086858 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:20.370469 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:20.370433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:21.156852 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:21.156813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:26.947256 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:26.947219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:27.045834 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:27.045798 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:27.144114 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:27.144076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:27.816549 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:27.816511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:27.919155 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:27.919114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:28.387947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:28.387902 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:52.419054 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:52.419018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:52.520220 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:52.520176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:48:59.506552 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:48:59.506446 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:05.617448 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:05.617406 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:06.035680 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:06.035653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:49:06.038760 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:06.038737 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:49:06.044551 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:06.044527 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:49:06.047539 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:06.047522 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:49:11.278672 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:11.278640 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:11.341713 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:11.341679 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:41.996527 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:41.996498 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:47.740211 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:47.740178 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:53.203976 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:53.203941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:53.304096 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:53.304058 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:53.452212 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:53.452173 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:53.773241 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:53.773208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:54.080606 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:54.080500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:56.847587 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:56.847543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:56.980321 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:56.980286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:57.000361 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000328 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:49:57.000689 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000677 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" containerName="cleanup" Jul 09 16:49:57.000735 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000690 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" containerName="cleanup" Jul 09 16:49:57.000735 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000697 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" containerName="maas-api" Jul 09 16:49:57.000735 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000703 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" containerName="maas-api" Jul 09 16:49:57.000829 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000777 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c0e142-810e-4a8b-a5ac-36d1d2e6b34b" containerName="maas-api" Jul 09 16:49:57.000829 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.000785 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdf81f1-b12b-4658-a3fa-7fe8160df94a" containerName="cleanup" Jul 09 16:49:57.003882 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.003861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.006669 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.006637 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-857c916f-serving-cert\"" Jul 09 16:49:57.006786 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.006674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2fl4\"" Jul 09 16:49:57.016133 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.016103 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:49:57.067004 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.066968 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:57.142238 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.142132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.142238 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.142195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jch4m\" (UniqueName: \"kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.142238 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.142235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.164729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.164694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:57.243022 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.242986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.243204 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.243042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch4m\" (UniqueName: \"kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.243204 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.243080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.243696 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.243671 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.245711 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.245691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.251509 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.251484 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch4m\" (UniqueName: \"kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m\") pod \"maas-api-e2e-mt-857c916f-79b9f64745-rf87k\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.315305 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.315269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:57.458336 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.458308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:49:57.460322 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:49:57.460295 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc817a8b_bf87_458b_8937_d7f9013d8f75.slice/crio-7af3fa577ae0aafc38bb66476e3d02b53946359f76eb41691f9306a17d7be86c WatchSource:0}: Error finding container 7af3fa577ae0aafc38bb66476e3d02b53946359f76eb41691f9306a17d7be86c: Status 404 returned error can't find the container with id 7af3fa577ae0aafc38bb66476e3d02b53946359f76eb41691f9306a17d7be86c Jul 09 16:49:57.500732 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.500696 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:57.601502 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.601468 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:49:57.912538 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:57.912503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" event={"ID":"fc817a8b-bf87-458b-8937-d7f9013d8f75","Type":"ContainerStarted","Data":"7af3fa577ae0aafc38bb66476e3d02b53946359f76eb41691f9306a17d7be86c"} Jul 09 16:49:58.921625 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:58.921586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" event={"ID":"fc817a8b-bf87-458b-8937-d7f9013d8f75","Type":"ContainerStarted","Data":"55ba9a0eb351d8976ab009f22a595cd0fb13bacceffe0726a017fa41a0990f34"} Jul 09 16:49:58.922029 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:58.921654 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:49:58.938739 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:49:58.938689 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" podStartSLOduration=2.223937571 podStartE2EDuration="2.938670482s" totalImagesPullingTime="714.732911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:58.937165842 +0000 UTC m=+1853.461490175" watchObservedRunningTime="2026-07-09 16:49:58.938670482 +0000 UTC m=+1853.462994814" Jul 09 16:50:01.295826 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:01.295787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:01.958498 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:01.958460 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:02.058897 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:02.058859 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:02.382325 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:02.382291 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:04.934333 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:04.934299 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:50:07.991414 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:07.991381 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:08.047527 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.047494 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:08.419609 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.419515 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:08.749380 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.749345 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:50:08.749638 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.749570 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" podUID="fc817a8b-bf87-458b-8937-d7f9013d8f75" containerName="maas-api" containerID="cri-o://55ba9a0eb351d8976ab009f22a595cd0fb13bacceffe0726a017fa41a0990f34" gracePeriod=30 Jul 09 16:50:08.773687 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.773658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:08.829503 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.829457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:08.973255 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.973224 2577 generic.go:356] "Generic (PLEG): container finished" podID="fc817a8b-bf87-458b-8937-d7f9013d8f75" containerID="55ba9a0eb351d8976ab009f22a595cd0fb13bacceffe0726a017fa41a0990f34" exitCode=0 Jul 09 16:50:08.973418 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:08.973285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" event={"ID":"fc817a8b-bf87-458b-8937-d7f9013d8f75","Type":"ContainerDied","Data":"55ba9a0eb351d8976ab009f22a595cd0fb13bacceffe0726a017fa41a0990f34"} Jul 09 16:50:09.001436 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.001368 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:50:09.130000 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.129946 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca\") pod \"fc817a8b-bf87-458b-8937-d7f9013d8f75\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " Jul 09 16:50:09.130183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.130021 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls\") pod \"fc817a8b-bf87-458b-8937-d7f9013d8f75\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " Jul 09 16:50:09.130183 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.130132 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jch4m\" (UniqueName: \"kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m\") pod \"fc817a8b-bf87-458b-8937-d7f9013d8f75\" (UID: \"fc817a8b-bf87-458b-8937-d7f9013d8f75\") " Jul 09 16:50:09.130386 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.130363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc817a8b-bf87-458b-8937-d7f9013d8f75" (UID: "fc817a8b-bf87-458b-8937-d7f9013d8f75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:50:09.130467 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.130452 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc817a8b-bf87-458b-8937-d7f9013d8f75-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:50:09.132280 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.132261 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc817a8b-bf87-458b-8937-d7f9013d8f75" (UID: "fc817a8b-bf87-458b-8937-d7f9013d8f75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:50:09.132363 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.132279 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m" (OuterVolumeSpecName: "kube-api-access-jch4m") pod "fc817a8b-bf87-458b-8937-d7f9013d8f75" (UID: "fc817a8b-bf87-458b-8937-d7f9013d8f75"). InnerVolumeSpecName "kube-api-access-jch4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:50:09.231508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.231469 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jch4m\" (UniqueName: \"kubernetes.io/projected/fc817a8b-bf87-458b-8937-d7f9013d8f75-kube-api-access-jch4m\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:50:09.231508 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.231503 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc817a8b-bf87-458b-8937-d7f9013d8f75-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:50:09.979198 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.979165 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" Jul 09 16:50:09.979374 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.979203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k" event={"ID":"fc817a8b-bf87-458b-8937-d7f9013d8f75","Type":"ContainerDied","Data":"7af3fa577ae0aafc38bb66476e3d02b53946359f76eb41691f9306a17d7be86c"} Jul 09 16:50:09.979374 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:09.979269 2577 scope.go:117] "RemoveContainer" containerID="55ba9a0eb351d8976ab009f22a595cd0fb13bacceffe0726a017fa41a0990f34" Jul 09 16:50:10.000466 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:10.000438 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:50:10.004225 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:10.004200 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-857c916f-79b9f64745-rf87k"] Jul 09 16:50:11.994754 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:11.994710 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc817a8b-bf87-458b-8937-d7f9013d8f75" path="/var/lib/kubelet/pods/fc817a8b-bf87-458b-8937-d7f9013d8f75/volumes" Jul 09 16:50:14.518561 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:14.518524 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:14.969517 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:14.969430 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:15.147093 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:15.147057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:15.239253 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:15.239227 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:15.379153 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:15.379100 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:21.163528 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:21.163493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:22.550002 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:22.549962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:27.495351 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:27.495267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:27.596857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:27.596803 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:34.221764 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:34.221725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:34.542351 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:34.542309 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:34.610906 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:34.610868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.039655 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.039623 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.139041 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.139006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.245355 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.245323 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:50:37.245761 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.245745 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc817a8b-bf87-458b-8937-d7f9013d8f75" containerName="maas-api" Jul 09 16:50:37.245761 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.245762 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc817a8b-bf87-458b-8937-d7f9013d8f75" containerName="maas-api" Jul 09 16:50:37.245886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.245833 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc817a8b-bf87-458b-8937-d7f9013d8f75" containerName="maas-api" Jul 09 16:50:37.249066 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.249037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.251782 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.251759 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-756fbf8e-serving-cert\"" Jul 09 16:50:37.251948 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.251828 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2fl4\"" Jul 09 16:50:37.258956 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.258910 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:50:37.277533 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.277496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.385908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.385816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6djm\" (UniqueName: \"kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.385908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.385886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.386122 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.386005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.445055 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.445003 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.486856 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.486809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.487058 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.486905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.487058 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.487014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6djm\" (UniqueName: \"kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.487538 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.487521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.489575 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.489556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.495694 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.495662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6djm\" (UniqueName: \"kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm\") pod \"maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.563514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.563475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:37.706714 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.706687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:50:37.709037 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:50:37.709009 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47857965_bab3_41c1_9ba2_2e227e717229.slice/crio-735dec9a997965395a911e466b820e9dfafaf99b364f03c7c6db0f18216c6a35 WatchSource:0}: Error finding container 735dec9a997965395a911e466b820e9dfafaf99b364f03c7c6db0f18216c6a35: Status 404 returned error can't find the container with id 735dec9a997965395a911e466b820e9dfafaf99b364f03c7c6db0f18216c6a35 Jul 09 16:50:37.710407 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.710387 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:50:37.831194 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.831163 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:37.927641 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:37.927538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:38.123980 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:38.123941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" event={"ID":"47857965-bab3-41c1-9ba2-2e227e717229","Type":"ContainerStarted","Data":"735dec9a997965395a911e466b820e9dfafaf99b364f03c7c6db0f18216c6a35"} Jul 09 16:50:39.130295 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:39.130254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" event={"ID":"47857965-bab3-41c1-9ba2-2e227e717229","Type":"ContainerStarted","Data":"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303"} Jul 09 16:50:39.130722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:39.130333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:39.148637 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:39.148584 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" podStartSLOduration=1.688824772 podStartE2EDuration="2.14856825s" totalImagesPullingTime="459.743478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:50:39.146219256 +0000 UTC m=+1893.670543588" watchObservedRunningTime="2026-07-09 16:50:39.14856825 +0000 UTC m=+1893.672892652" Jul 09 16:50:41.437848 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:41.437812 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:50:45.141388 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:45.141359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:50:53.283645 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:50:53.283610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:51:39.703961 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:39.703902 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:51:39.704534 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:39.704189 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" podUID="47857965-bab3-41c1-9ba2-2e227e717229" containerName="maas-api" containerID="cri-o://077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303" gracePeriod=30 Jul 09 16:51:39.963979 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:39.963905 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:51:40.022970 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.022936 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:51:40.074363 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.074333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:51:40.137579 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.137546 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca\") pod \"47857965-bab3-41c1-9ba2-2e227e717229\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " Jul 09 16:51:40.137789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.137601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls\") pod \"47857965-bab3-41c1-9ba2-2e227e717229\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " Jul 09 16:51:40.137789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.137731 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6djm\" (UniqueName: \"kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm\") pod \"47857965-bab3-41c1-9ba2-2e227e717229\" (UID: \"47857965-bab3-41c1-9ba2-2e227e717229\") " Jul 09 16:51:40.138012 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.137981 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47857965-bab3-41c1-9ba2-2e227e717229" (UID: "47857965-bab3-41c1-9ba2-2e227e717229"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:51:40.138138 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.138118 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47857965-bab3-41c1-9ba2-2e227e717229-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:51:40.140051 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.140018 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm" (OuterVolumeSpecName: "kube-api-access-z6djm") pod "47857965-bab3-41c1-9ba2-2e227e717229" (UID: "47857965-bab3-41c1-9ba2-2e227e717229"). InnerVolumeSpecName "kube-api-access-z6djm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:51:40.140150 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.140106 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47857965-bab3-41c1-9ba2-2e227e717229" (UID: "47857965-bab3-41c1-9ba2-2e227e717229"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:51:40.238936 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.238888 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6djm\" (UniqueName: \"kubernetes.io/projected/47857965-bab3-41c1-9ba2-2e227e717229-kube-api-access-z6djm\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:51:40.238936 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.238931 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47857965-bab3-41c1-9ba2-2e227e717229-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:51:40.434821 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.434781 2577 generic.go:356] "Generic (PLEG): container finished" podID="47857965-bab3-41c1-9ba2-2e227e717229" containerID="077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303" exitCode=0 Jul 09 16:51:40.434821 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.434824 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" Jul 09 16:51:40.435081 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.434874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" event={"ID":"47857965-bab3-41c1-9ba2-2e227e717229","Type":"ContainerDied","Data":"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303"} Jul 09 16:51:40.435081 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.434951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs" event={"ID":"47857965-bab3-41c1-9ba2-2e227e717229","Type":"ContainerDied","Data":"735dec9a997965395a911e466b820e9dfafaf99b364f03c7c6db0f18216c6a35"} Jul 09 16:51:40.435081 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.434976 2577 scope.go:117] "RemoveContainer" containerID="077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303" Jul 09 16:51:40.444119 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.444104 2577 scope.go:117] "RemoveContainer" containerID="077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303" Jul 09 16:51:40.444377 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:51:40.444359 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303\": container with ID starting with 077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303 not found: ID does not exist" containerID="077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303" Jul 09 16:51:40.444422 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.444386 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303"} err="failed to get container status \"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303\": rpc error: code = NotFound desc = could not find container \"077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303\": container with ID starting with 077acc7b526ea1c76b108e7b99c162b24a847383531a5480c2ba6c1794856303 not found: ID does not exist" Jul 09 16:51:40.455812 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.455785 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:51:40.459831 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:40.459808 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-756fbf8e-84464f9bc7-7hvrs"] Jul 09 16:51:41.993025 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:41.992992 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47857965-bab3-41c1-9ba2-2e227e717229" path="/var/lib/kubelet/pods/47857965-bab3-41c1-9ba2-2e227e717229/volumes" Jul 09 16:51:45.422010 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:45.421974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:51:50.511724 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:51:50.511688 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:26.003741 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:26.003708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:27.476531 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:27.476502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:27.604681 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:27.604652 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:33.103262 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:33.103222 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:33.204168 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:33.204135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:33.367030 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:33.366945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:33.607854 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:33.607821 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:34.850521 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:34.850484 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:34.937413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:34.937367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:35.037137 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:35.037102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:35.718374 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:35.718337 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:35.817437 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:35.817400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:52:35.872911 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:52:35.872880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:07.163409 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:07.163367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:12.724198 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:12.724162 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:26.829112 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:26.829019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:26.886207 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:26.886157 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:53.187057 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:53.187020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:53.286389 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:53.286354 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:53:57.405692 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:53:57.405655 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:03.291303 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:03.291268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:06.065857 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:06.065823 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:54:06.069824 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:06.069796 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:54:06.074442 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:06.074425 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:54:06.078564 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:06.078547 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:54:09.156422 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:09.156386 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:09.258623 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:09.258591 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:19.442726 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:19.442684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:19.766276 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:19.766239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:40.705507 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:40.705471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:46.348018 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:46.347982 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:46.448453 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:46.448412 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:46.674419 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:46.674336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:46.979662 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:46.979634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:47.084283 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:47.084226 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.192765 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.192730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.267588 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.267548 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:54:50.268239 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.268213 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47857965-bab3-41c1-9ba2-2e227e717229" containerName="maas-api" Jul 09 16:54:50.268239 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.268237 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47857965-bab3-41c1-9ba2-2e227e717229" containerName="maas-api" Jul 09 16:54:50.268452 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.268347 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47857965-bab3-41c1-9ba2-2e227e717229" containerName="maas-api" Jul 09 16:54:50.270153 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.270127 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.271802 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.271782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.274554 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.274531 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9d86cf-serving-cert\"" Jul 09 16:54:50.274659 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.274534 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2fl4\"" Jul 09 16:54:50.282597 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.282571 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:54:50.341161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.341091 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.414180 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.414147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bv7b\" (UniqueName: \"kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.414368 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.414194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.414368 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.414237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.449329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.449223 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.515266 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.515229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.515440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.515322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bv7b\" (UniqueName: \"kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.515440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.515354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.516064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.516038 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.517963 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.517904 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.523815 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.523787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bv7b\" (UniqueName: \"kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b\") pod \"maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.585201 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.585168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:50.780161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.780125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.881015 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.880979 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:50.936619 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:50.936593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:54:51.359520 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:51.359478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" event={"ID":"2459c6b2-707c-40c6-a1a9-ed59bd3d3742","Type":"ContainerStarted","Data":"a307492f322c8845e6b1374b1a811a74ee21331e729bf59912d1c79c364748e2"} Jul 09 16:54:52.365755 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:52.365721 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" event={"ID":"2459c6b2-707c-40c6-a1a9-ed59bd3d3742","Type":"ContainerStarted","Data":"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1"} Jul 09 16:54:52.366156 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:52.365853 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:52.384351 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:52.384295 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" podStartSLOduration=1.828658068 podStartE2EDuration="2.384276735s" totalImagesPullingTime="555.618667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:54:52.383210476 +0000 UTC m=+2146.907534808" watchObservedRunningTime="2026-07-09 16:54:52.384276735 +0000 UTC m=+2146.908601066" Jul 09 16:54:53.753307 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:53.753269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:53.854222 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:53.854191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:55.084304 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.084272 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:55.144419 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.144390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:55.265833 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.265779 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 16:54:55.269719 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.269693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.272331 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.272303 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1876d0-openshift-default-dockercfg-z2mld\"" Jul 09 16:54:55.272457 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.272307 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 16:54:55.287266 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.287233 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 16:54:55.323577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.323529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:55.359130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359130 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585z9\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.359402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.359332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.459791 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.459753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.459997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.459833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.459997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.459881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.459997 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.459962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460160 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460160 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585z9\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460160 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460160 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.460768 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.460744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.462407 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.462381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.462874 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.462856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.467676 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.467653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-585z9\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.467974 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.467954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token\") pod \"e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.582531 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.582500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:55.723550 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:55.723520 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 16:54:55.725715 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:54:55.725685 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658dfc2a_adb5_4d91_8528_b917ec64c731.slice/crio-090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b WatchSource:0}: Error finding container 090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b: Status 404 returned error can't find the container with id 090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b Jul 09 16:54:56.060045 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:56.060001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:54:56.392802 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:56.392715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerStarted","Data":"090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b"} Jul 09 16:54:58.382910 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:58.382881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:54:58.469722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:58.469681 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:54:58.469823 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:58.469784 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:54:58.469823 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:58.469817 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:54:59.411029 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:59.410994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerStarted","Data":"371adc5b9a45be4238daea1593d5caf9cf01b1ee474be55b7392eda15f42977a"} Jul 09 16:54:59.433204 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:59.433158 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podStartSLOduration=1.692186268 podStartE2EDuration="4.433141985s" totalImagesPullingTime="2.740955717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:54:59.431001971 +0000 UTC m=+2153.955326302" watchObservedRunningTime="2026-07-09 16:54:59.433141985 +0000 UTC m=+2153.957466316" Jul 09 16:54:59.583182 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:59.583142 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:54:59.588417 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:54:59.588389 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:00.417212 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:00.417172 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:00.418285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:00.418265 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:01.365398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.365361 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 16:55:01.370070 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.370041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.372855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.372826 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1876d0-serving-cert\"" Jul 09 16:55:01.378599 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.378509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 16:55:01.517616 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.517577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.518124 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.517890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9h45\" (UniqueName: \"kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.518124 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.517984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.619064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.618960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.619064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.619036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9h45\" (UniqueName: \"kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.619064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.619063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.619356 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:55:01.619094 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-serving-cert: secret "maas-api-e2e-inf-b-1876d0-serving-cert" not found Jul 09 16:55:01.619356 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:55:01.619185 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls podName:4664e641-42bc-42c2-a56b-0ecf320c409a nodeName:}" failed. No retries permitted until 2026-07-09 16:55:02.119161788 +0000 UTC m=+2156.643486097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls") pod "maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" (UID: "4664e641-42bc-42c2-a56b-0ecf320c409a") : secret "maas-api-e2e-inf-b-1876d0-serving-cert" not found Jul 09 16:55:01.619746 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.619727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.627762 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.627726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9h45\" (UniqueName: \"kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:01.646034 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.645993 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:01.794227 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.794191 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:01.910190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:01.910105 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:02.009717 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.009680 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:02.123237 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.123204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:02.126641 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.126612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") pod \"maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:02.132151 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.132123 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:02.283892 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.283843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:02.432632 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:02.432605 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 16:55:02.434554 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:55:02.434523 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4664e641_42bc_42c2_a56b_0ecf320c409a.slice/crio-f93be731161740999415b1da922598ae764285f96670e93729f74d3557df52c8 WatchSource:0}: Error finding container f93be731161740999415b1da922598ae764285f96670e93729f74d3557df52c8: Status 404 returned error can't find the container with id f93be731161740999415b1da922598ae764285f96670e93729f74d3557df52c8 Jul 09 16:55:03.434431 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:03.434394 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" event={"ID":"4664e641-42bc-42c2-a56b-0ecf320c409a","Type":"ContainerStarted","Data":"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4"} Jul 09 16:55:03.434947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:03.434451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" event={"ID":"4664e641-42bc-42c2-a56b-0ecf320c409a","Type":"ContainerStarted","Data":"f93be731161740999415b1da922598ae764285f96670e93729f74d3557df52c8"} Jul 09 16:55:03.434947 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:03.434498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:03.453138 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:03.453090 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" podStartSLOduration=2.044545468 podStartE2EDuration="2.453073747s" totalImagesPullingTime="408.528279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:55:03.451589249 +0000 UTC m=+2157.975913580" watchObservedRunningTime="2026-07-09 16:55:03.453073747 +0000 UTC m=+2157.977398078" Jul 09 16:55:05.760753 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:05.760716 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:09.445179 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:09.445149 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 16:55:11.561646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:11.561604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:11.684112 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:11.684081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:15.419059 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.419023 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 16:55:15.419465 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.419093 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 16:55:15.495282 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.495248 2577 generic.go:356] "Generic (PLEG): container finished" podID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerID="371adc5b9a45be4238daea1593d5caf9cf01b1ee474be55b7392eda15f42977a" exitCode=0 Jul 09 16:55:15.495460 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.495323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerDied","Data":"371adc5b9a45be4238daea1593d5caf9cf01b1ee474be55b7392eda15f42977a"} Jul 09 16:55:15.495723 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.495709 2577 scope.go:117] "RemoveContainer" containerID="371adc5b9a45be4238daea1593d5caf9cf01b1ee474be55b7392eda15f42977a" Jul 09 16:55:15.496714 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.496686 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:55:15.496766 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.496754 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:55:15.496811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:15.496787 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:55:16.506693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:16.506554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerStarted","Data":"66c61e97f13ee03c464eb9f8d5856483c68415a3da75d7592d600dc879400201"} Jul 09 16:55:16.583161 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:16.583117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:16.583477 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:16.583449 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 16:55:16.583589 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:16.583551 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 16:55:17.583476 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:17.583431 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 16:55:17.583879 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:17.583489 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 16:55:18.583274 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:18.583233 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 16:55:18.583486 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:18.583300 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 16:55:19.583314 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:19.583276 2577 patch_prober.go:28] interesting pod/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 16:55:19.583715 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:19.583334 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 16:55:20.325602 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:20.325570 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:20.591646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:20.591560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:20.592078 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:20.591837 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:20.592628 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:20.592607 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 16:55:22.783342 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:22.783306 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:24.219887 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:24.219851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:55:50.656196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:55:50.656160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:08.869378 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:08.869347 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:09.117364 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.117329 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:56:09.121067 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.121007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.124075 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.124053 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-7ab20f-openshift-default-dockercfg-lwwsl\"" Jul 09 16:56:09.131650 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.131626 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:56:09.144357 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.144317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:09.261494 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261593 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc6q\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.261708 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.262118 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.261742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.362986 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.362941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.362997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc6q\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363143 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363196 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363557 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363557 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363557 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363934 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363892 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.363999 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.363950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.365419 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.365400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.365693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.365675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.372337 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.372267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.372689 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.372664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc6q\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.432239 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.432200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:09.493295 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.493267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:09.580068 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.579940 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:56:09.582106 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:56:09.582066 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8597e24_328c_480a_a0d1_4af7c5883198.slice/crio-0086b7e2668e51e72ddefb4e5d75c18290571f6dbb24cbef00b18ddc1f4ff333 WatchSource:0}: Error finding container 0086b7e2668e51e72ddefb4e5d75c18290571f6dbb24cbef00b18ddc1f4ff333: Status 404 returned error can't find the container with id 0086b7e2668e51e72ddefb4e5d75c18290571f6dbb24cbef00b18ddc1f4ff333 Jul 09 16:56:09.584601 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.584557 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:09.584679 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.584661 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:09.584719 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.584707 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:09.779153 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.779111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerStarted","Data":"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782"} Jul 09 16:56:09.779329 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.779178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerStarted","Data":"0086b7e2668e51e72ddefb4e5d75c18290571f6dbb24cbef00b18ddc1f4ff333"} Jul 09 16:56:09.798393 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:09.798337 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podStartSLOduration=0.798317203 podStartE2EDuration="798.317203ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:09.796629392 +0000 UTC m=+2224.320953724" watchObservedRunningTime="2026-07-09 16:56:09.798317203 +0000 UTC m=+2224.322641534" Jul 09 16:56:10.432844 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:10.432806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:10.438052 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:10.438023 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:10.785221 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:10.785189 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:10.786342 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:10.786317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:11.736426 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.736378 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:11.833902 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.833856 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:11.857076 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.857036 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:11.860833 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.860809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:11.863644 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.863621 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-reuse-7ab20f-serving-cert\"" Jul 09 16:56:11.871065 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.871042 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:11.984464 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.984433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:11.986740 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.986656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bg6c\" (UniqueName: \"kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:11.986740 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.986700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:11.986912 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:11.986776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.084737 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.084705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:12.087949 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.087903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bg6c\" (UniqueName: \"kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.088019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.087967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.088072 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.088018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.088719 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.088698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.090744 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.090714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.097075 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.096865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bg6c\" (UniqueName: \"kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c\") pod \"maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.173211 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.173165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.317877 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.317849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:12.320577 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:56:12.320546 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccac1c26_b02d_469b_8662_a22c0954d032.slice/crio-50d70f90b917eaddad740afdb53d560e184a7917129af3f66e32f5fe86d8191f WatchSource:0}: Error finding container 50d70f90b917eaddad740afdb53d560e184a7917129af3f66e32f5fe86d8191f: Status 404 returned error can't find the container with id 50d70f90b917eaddad740afdb53d560e184a7917129af3f66e32f5fe86d8191f Jul 09 16:56:12.322632 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.322609 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:56:12.408343 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.408305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:12.508565 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.508529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:12.797894 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.797860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" event={"ID":"ccac1c26-b02d-469b-8662-a22c0954d032","Type":"ContainerStarted","Data":"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428"} Jul 09 16:56:12.798334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.797957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" event={"ID":"ccac1c26-b02d-469b-8662-a22c0954d032","Type":"ContainerStarted","Data":"50d70f90b917eaddad740afdb53d560e184a7917129af3f66e32f5fe86d8191f"} Jul 09 16:56:12.798334 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.798122 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:12.816390 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:12.816339 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" podStartSLOduration=1.403191571 podStartE2EDuration="1.816319166s" totalImagesPullingTime="413.127595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:12.813690966 +0000 UTC m=+2227.338015296" watchObservedRunningTime="2026-07-09 16:56:12.816319166 +0000 UTC m=+2227.340643502" Jul 09 16:56:16.227755 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:16.227710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:18.812961 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:18.812896 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:25.786505 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.786470 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 16:56:25.786954 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.786527 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 16:56:25.874872 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.874829 2577 generic.go:356] "Generic (PLEG): container finished" podID="d8597e24-328c-480a-a0d1-4af7c5883198" containerID="f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782" exitCode=0 Jul 09 16:56:25.875050 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.874882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerDied","Data":"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782"} Jul 09 16:56:25.875301 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.875288 2577 scope.go:117] "RemoveContainer" containerID="f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782" Jul 09 16:56:25.876371 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.876340 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:25.876440 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.876423 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:25.876489 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:25.876474 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:56:26.881544 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:26.881470 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerStarted","Data":"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5"} Jul 09 16:56:27.433315 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:27.433270 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:27.433590 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:27.433565 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 16:56:27.433683 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:27.433623 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 16:56:28.432653 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:28.432607 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 16:56:28.433244 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:28.432687 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 16:56:29.433134 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:29.433095 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 16:56:29.433531 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:29.433154 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 16:56:30.433187 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:30.433150 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 16:56:30.433582 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:30.433206 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 16:56:31.436895 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:31.436867 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:31.437333 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:31.437141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:31.437888 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:31.437870 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:56:33.738211 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:33.738175 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:57.362693 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.362654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:57.696986 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.696884 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:57.697175 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.697151 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" podUID="ccac1c26-b02d-469b-8662-a22c0954d032" containerName="maas-api" containerID="cri-o://c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428" gracePeriod=30 Jul 09 16:56:57.715539 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.715519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:57.813314 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.813280 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:56:57.958355 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:57.958282 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:58.029752 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.029727 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca\") pod \"ccac1c26-b02d-469b-8662-a22c0954d032\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " Jul 09 16:56:58.029986 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.029809 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls\") pod \"ccac1c26-b02d-469b-8662-a22c0954d032\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " Jul 09 16:56:58.029986 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.029858 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bg6c\" (UniqueName: \"kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c\") pod \"ccac1c26-b02d-469b-8662-a22c0954d032\" (UID: \"ccac1c26-b02d-469b-8662-a22c0954d032\") " Jul 09 16:56:58.030454 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.030332 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccac1c26-b02d-469b-8662-a22c0954d032" (UID: "ccac1c26-b02d-469b-8662-a22c0954d032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:56:58.032402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.032372 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccac1c26-b02d-469b-8662-a22c0954d032" (UID: "ccac1c26-b02d-469b-8662-a22c0954d032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:56:58.032503 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.032422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c" (OuterVolumeSpecName: "kube-api-access-2bg6c") pod "ccac1c26-b02d-469b-8662-a22c0954d032" (UID: "ccac1c26-b02d-469b-8662-a22c0954d032"). InnerVolumeSpecName "kube-api-access-2bg6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:58.048427 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.048396 2577 generic.go:356] "Generic (PLEG): container finished" podID="ccac1c26-b02d-469b-8662-a22c0954d032" containerID="c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428" exitCode=0 Jul 09 16:56:58.048559 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.048446 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" Jul 09 16:56:58.048559 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.048485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" event={"ID":"ccac1c26-b02d-469b-8662-a22c0954d032","Type":"ContainerDied","Data":"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428"} Jul 09 16:56:58.048740 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.048561 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2" event={"ID":"ccac1c26-b02d-469b-8662-a22c0954d032","Type":"ContainerDied","Data":"50d70f90b917eaddad740afdb53d560e184a7917129af3f66e32f5fe86d8191f"} Jul 09 16:56:58.048740 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.048594 2577 scope.go:117] "RemoveContainer" containerID="c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428" Jul 09 16:56:58.059228 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.059209 2577 scope.go:117] "RemoveContainer" containerID="c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428" Jul 09 16:56:58.059537 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:56:58.059518 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428\": container with ID starting with c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428 not found: ID does not exist" containerID="c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428" Jul 09 16:56:58.059594 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.059548 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428"} err="failed to get container status \"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428\": rpc error: code = NotFound desc = could not find container \"c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428\": container with ID starting with c53a17f8aff2adb51da726e76784980507515e8763149edcd8802009ef7cc428 not found: ID does not exist" Jul 09 16:56:58.069687 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.069662 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:58.073595 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.073569 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-reuse-7ab20f-59b4874d9f-6c7p2"] Jul 09 16:56:58.130659 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.130624 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccac1c26-b02d-469b-8662-a22c0954d032-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:56:58.130659 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.130656 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bg6c\" (UniqueName: \"kubernetes.io/projected/ccac1c26-b02d-469b-8662-a22c0954d032-kube-api-access-2bg6c\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:56:58.130659 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:58.130666 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac1c26-b02d-469b-8662-a22c0954d032-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:56:59.993934 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:56:59.993879 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccac1c26-b02d-469b-8662-a22c0954d032" path="/var/lib/kubelet/pods/ccac1c26-b02d-469b-8662-a22c0954d032/volumes" Jul 09 16:57:02.778240 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:02.778197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:15.955027 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:15.954984 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:38.917580 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:38.917543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:39.808210 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:39.808174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:45.316398 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.316362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:45.420100 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.420063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:45.577646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.577552 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:57:45.577896 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.577858 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" containerID="cri-o://6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5" gracePeriod=30 Jul 09 16:57:45.598612 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.598564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:45.841927 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:45.841817 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:46.189939 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.189808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:46.440089 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.439998 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 16:57:46.440448 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.440076 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 16:57:46.468152 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.468116 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:57:46.468762 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.468745 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccac1c26-b02d-469b-8662-a22c0954d032" containerName="maas-api" Jul 09 16:57:46.468762 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.468764 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccac1c26-b02d-469b-8662-a22c0954d032" containerName="maas-api" Jul 09 16:57:46.468948 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.468850 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccac1c26-b02d-469b-8662-a22c0954d032" containerName="maas-api" Jul 09 16:57:46.472788 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.472751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.475592 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.475568 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-7ab20f-openshift-default-dockercfg-5znmn\"" Jul 09 16:57:46.482600 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.482273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:57:46.493385 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.493346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:46.565080 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565080 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565402 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565563 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxrm\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565563 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565675 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.565675 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.565654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666301 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666443 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666443 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666443 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666603 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxrm\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666754 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666754 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666789 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.666860 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.666835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.667084 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.667059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.667392 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.667375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.669158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.669129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.669282 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.669257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.678491 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.678465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxrm\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.678715 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.678699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token\") pod \"e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.707033 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.706941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:46.786123 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.786089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:46.930237 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.930024 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:57:46.933130 ip-10-0-133-162 kubenswrapper[2577]: W0709 16:57:46.933103 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7595d21b_637c_4ff6_822a_af7c1de09375.slice/crio-b193ad0167b3e139c0aa1e03ebb02ec662b7c4b0b2e229f3b17d37fc66f92694 WatchSource:0}: Error finding container b193ad0167b3e139c0aa1e03ebb02ec662b7c4b0b2e229f3b17d37fc66f92694: Status 404 returned error can't find the container with id b193ad0167b3e139c0aa1e03ebb02ec662b7c4b0b2e229f3b17d37fc66f92694 Jul 09 16:57:46.935615 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.935566 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:57:46.935710 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.935657 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:57:46.935751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:46.935706 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:57:47.292732 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:47.292697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerStarted","Data":"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e"} Jul 09 16:57:47.292936 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:47.292756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerStarted","Data":"b193ad0167b3e139c0aa1e03ebb02ec662b7c4b0b2e229f3b17d37fc66f92694"} Jul 09 16:57:47.313729 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:47.313680 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podStartSLOduration=1.313664808 podStartE2EDuration="1.313664808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:57:47.311467443 +0000 UTC m=+2321.835791777" watchObservedRunningTime="2026-07-09 16:57:47.313664808 +0000 UTC m=+2321.837989140" Jul 09 16:57:47.787004 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:47.786960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:47.792097 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:47.792070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:48.298103 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.298060 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:48.299093 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.299070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:57:48.366218 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.366181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:48.445557 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.445521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:48.524774 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.524726 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:48.837951 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:48.837899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:57:50.844804 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.844739 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:57:50.907886 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.907851 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.907903 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.907951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908008 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908076 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908108 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908171 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908147 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908302 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908204 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npc6q\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908302 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908226 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data\") pod \"d8597e24-328c-480a-a0d1-4af7c5883198\" (UID: \"d8597e24-328c-480a-a0d1-4af7c5883198\") " Jul 09 16:57:50.908413 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908370 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:57:50.908553 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908509 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:57:50.908631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908570 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:57:50.908631 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908600 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:57:50.908734 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908670 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:50.908734 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.908692 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:50.909685 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.909661 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:57:50.910675 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.910650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:57:50.910785 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.910771 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q" (OuterVolumeSpecName: "kube-api-access-npc6q") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "kube-api-access-npc6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:57:50.910835 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.910777 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:57:50.910835 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:50.910771 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8597e24-328c-480a-a0d1-4af7c5883198" (UID: "d8597e24-328c-480a-a0d1-4af7c5883198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:57:51.009958 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.009897 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.009958 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.009958 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8597e24-328c-480a-a0d1-4af7c5883198-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.010163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.009973 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npc6q\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-kube-api-access-npc6q\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.010163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.009987 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.010163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.010001 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8597e24-328c-480a-a0d1-4af7c5883198-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.010163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.010011 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8597e24-328c-480a-a0d1-4af7c5883198-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.010163 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.010019 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8597e24-328c-480a-a0d1-4af7c5883198-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:57:51.316394 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.316361 2577 generic.go:356] "Generic (PLEG): container finished" podID="d8597e24-328c-480a-a0d1-4af7c5883198" containerID="6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5" exitCode=0 Jul 09 16:57:51.316577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.316414 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" Jul 09 16:57:51.316577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.316449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerDied","Data":"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5"} Jul 09 16:57:51.316577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.316505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh" event={"ID":"d8597e24-328c-480a-a0d1-4af7c5883198","Type":"ContainerDied","Data":"0086b7e2668e51e72ddefb4e5d75c18290571f6dbb24cbef00b18ddc1f4ff333"} Jul 09 16:57:51.316577 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.316536 2577 scope.go:117] "RemoveContainer" containerID="6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5" Jul 09 16:57:51.326344 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.326326 2577 scope.go:117] "RemoveContainer" containerID="f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782" Jul 09 16:57:51.335162 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.335141 2577 scope.go:117] "RemoveContainer" containerID="6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5" Jul 09 16:57:51.335451 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:57:51.335428 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5\": container with ID starting with 6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5 not found: ID does not exist" containerID="6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5" Jul 09 16:57:51.335514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.335461 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5"} err="failed to get container status \"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5\": rpc error: code = NotFound desc = could not find container \"6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5\": container with ID starting with 6c0c05dbcce2acf308ce82816abc8858a2eb83eda1a02840d4242595c56dacc5 not found: ID does not exist" Jul 09 16:57:51.335514 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.335480 2577 scope.go:117] "RemoveContainer" containerID="f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782" Jul 09 16:57:51.335748 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:57:51.335727 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782\": container with ID starting with f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782 not found: ID does not exist" containerID="f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782" Jul 09 16:57:51.335820 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.335756 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782"} err="failed to get container status \"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782\": rpc error: code = NotFound desc = could not find container \"f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782\": container with ID starting with f2e9a9deb92c4b3ad96b10f0b27912b8fe2a90a764f813ebed8e3853d7237782 not found: ID does not exist" Jul 09 16:57:51.337420 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.337396 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:57:51.340459 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.340438 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-9strh"] Jul 09 16:57:51.993698 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:51.993665 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" path="/var/lib/kubelet/pods/d8597e24-328c-480a-a0d1-4af7c5883198/volumes" Jul 09 16:57:52.862164 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:57:52.862130 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:02.375365 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.375330 2577 generic.go:356] "Generic (PLEG): container finished" podID="7595d21b-637c-4ff6-822a-af7c1de09375" containerID="886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e" exitCode=0 Jul 09 16:58:02.375787 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.375409 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerDied","Data":"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e"} Jul 09 16:58:02.375832 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.375810 2577 scope.go:117] "RemoveContainer" containerID="886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e" Jul 09 16:58:02.377019 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.376985 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:58:02.377132 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.377074 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:58:02.377132 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:02.377123 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 16:58:03.298595 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:03.298550 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:58:03.382722 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:03.382690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerStarted","Data":"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338"} Jul 09 16:58:03.788592 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:03.788549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:58:03.788804 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:03.788739 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 09 16:58:03.788873 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:03.788798 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 09 16:58:04.787284 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:04.787243 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 09 16:58:04.787789 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:04.787311 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 09 16:58:05.457364 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:05.457335 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:05.786464 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:05.786430 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 09 16:58:05.786636 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:05.786489 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 09 16:58:06.786795 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:06.786756 2577 patch_prober.go:28] interesting pod/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 09 16:58:06.787285 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:06.786814 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 09 16:58:07.791078 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:07.791045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:58:07.791452 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:07.791314 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:58:07.792043 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:07.792022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:58:28.886853 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:28.886820 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:29.208620 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:29.208535 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:29.308855 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:29.308823 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:34.287247 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:34.287211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:54.058064 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:54.058032 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:54.159833 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:54.159801 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:56.975381 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:56.975349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:58:57.076647 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:58:57.076614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:06.097214 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:06.097185 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:59:06.101845 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:06.101823 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 16:59:06.105433 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:06.105405 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:59:06.110112 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:06.110093 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 16:59:11.050566 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:11.050531 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:11.315500 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:11.315409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:16.845473 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:16.845411 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:16.901190 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:16.901158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:17.129530 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:17.129431 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:59:17.129751 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:17.129718 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" containerID="cri-o://4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338" gracePeriod=30 Jul 09 16:59:17.204832 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:17.204799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:17.365657 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:17.365624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:18.230342 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:18.230310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.140246 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.140208 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.291040 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.290997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.685003 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.684962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.785878 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.785842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.886136 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.886101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:19.950382 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.950292 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:59:19.950562 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.950493 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" podUID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" containerName="maas-api" containerID="cri-o://b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1" gracePeriod=30 Jul 09 16:59:19.986226 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:19.986194 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:20.221802 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.221767 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:59:20.361563 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.361517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca\") pod \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " Jul 09 16:59:20.361972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.361589 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls\") pod \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " Jul 09 16:59:20.361972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.361709 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bv7b\" (UniqueName: \"kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b\") pod \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\" (UID: \"2459c6b2-707c-40c6-a1a9-ed59bd3d3742\") " Jul 09 16:59:20.361972 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.361906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2459c6b2-707c-40c6-a1a9-ed59bd3d3742" (UID: "2459c6b2-707c-40c6-a1a9-ed59bd3d3742"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:59:20.362176 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.362159 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:20.363884 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.363863 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b" (OuterVolumeSpecName: "kube-api-access-4bv7b") pod "2459c6b2-707c-40c6-a1a9-ed59bd3d3742" (UID: "2459c6b2-707c-40c6-a1a9-ed59bd3d3742"). InnerVolumeSpecName "kube-api-access-4bv7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:59:20.364009 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.363945 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2459c6b2-707c-40c6-a1a9-ed59bd3d3742" (UID: "2459c6b2-707c-40c6-a1a9-ed59bd3d3742"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:59:20.462953 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.462855 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bv7b\" (UniqueName: \"kubernetes.io/projected/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-kube-api-access-4bv7b\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:20.462953 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.462883 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459c6b2-707c-40c6-a1a9-ed59bd3d3742-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:20.795730 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.795697 2577 generic.go:356] "Generic (PLEG): container finished" podID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" containerID="b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1" exitCode=0 Jul 09 16:59:20.795908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.795743 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" Jul 09 16:59:20.795908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.795737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" event={"ID":"2459c6b2-707c-40c6-a1a9-ed59bd3d3742","Type":"ContainerDied","Data":"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1"} Jul 09 16:59:20.795908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.795873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd" event={"ID":"2459c6b2-707c-40c6-a1a9-ed59bd3d3742","Type":"ContainerDied","Data":"a307492f322c8845e6b1374b1a811a74ee21331e729bf59912d1c79c364748e2"} Jul 09 16:59:20.795908 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.795897 2577 scope.go:117] "RemoveContainer" containerID="b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1" Jul 09 16:59:20.812553 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.812526 2577 scope.go:117] "RemoveContainer" containerID="b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1" Jul 09 16:59:20.812855 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:59:20.812833 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1\": container with ID starting with b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1 not found: ID does not exist" containerID="b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1" Jul 09 16:59:20.812904 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.812865 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1"} err="failed to get container status \"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1\": rpc error: code = NotFound desc = could not find container \"b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1\": container with ID starting with b0e6b8cfe4ba7075acbbc98fb96bdfe0547a22f7e32ae01d507a56055884cad1 not found: ID does not exist" Jul 09 16:59:20.820141 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.820112 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:59:20.822251 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:20.822225 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d86cf-6bdc5db74b-mfwmd"] Jul 09 16:59:21.995074 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:21.995039 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" path="/var/lib/kubelet/pods/2459c6b2-707c-40c6-a1a9-ed59bd3d3742/volumes" Jul 09 16:59:22.390149 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.390119 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:59:22.481573 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481585 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481631 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481672 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481710 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.481771 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htxrm\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.482158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481814 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy\") pod \"7595d21b-637c-4ff6-822a-af7c1de09375\" (UID: \"7595d21b-637c-4ff6-822a-af7c1de09375\") " Jul 09 16:59:22.482158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.481901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:59:22.482158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482102 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:59:22.482158 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482143 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:59:22.482372 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482195 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.482372 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482218 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.482468 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482448 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:59:22.482663 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.482640 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data" (OuterVolumeSpecName: "istio-data") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:59:22.484335 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.484310 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:59:22.484581 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.484563 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:59:22.484763 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.484743 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm" (OuterVolumeSpecName: "kube-api-access-htxrm") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "kube-api-access-htxrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:59:22.484828 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.484770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token" (OuterVolumeSpecName: "istio-token") pod "7595d21b-637c-4ff6-822a-af7c1de09375" (UID: "7595d21b-637c-4ff6-822a-af7c1de09375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:59:22.582960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582858 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7595d21b-637c-4ff6-822a-af7c1de09375-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.582960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582885 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7595d21b-637c-4ff6-822a-af7c1de09375-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.582960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582894 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.582960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582902 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.582960 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582956 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.583216 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582967 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxrm\" (UniqueName: \"kubernetes.io/projected/7595d21b-637c-4ff6-822a-af7c1de09375-kube-api-access-htxrm\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.583216 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.582976 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7595d21b-637c-4ff6-822a-af7c1de09375-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 16:59:22.812646 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.812612 2577 generic.go:356] "Generic (PLEG): container finished" podID="7595d21b-637c-4ff6-822a-af7c1de09375" containerID="4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338" exitCode=0 Jul 09 16:59:22.812811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.812666 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" Jul 09 16:59:22.812811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.812692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerDied","Data":"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338"} Jul 09 16:59:22.812811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.812754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd" event={"ID":"7595d21b-637c-4ff6-822a-af7c1de09375","Type":"ContainerDied","Data":"b193ad0167b3e139c0aa1e03ebb02ec662b7c4b0b2e229f3b17d37fc66f92694"} Jul 09 16:59:22.812811 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.812777 2577 scope.go:117] "RemoveContainer" containerID="4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338" Jul 09 16:59:22.822976 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.822953 2577 scope.go:117] "RemoveContainer" containerID="886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e" Jul 09 16:59:22.831576 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.831554 2577 scope.go:117] "RemoveContainer" containerID="4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338" Jul 09 16:59:22.831878 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:59:22.831859 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338\": container with ID starting with 4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338 not found: ID does not exist" containerID="4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338" Jul 09 16:59:22.831943 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.831887 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338"} err="failed to get container status \"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338\": rpc error: code = NotFound desc = could not find container \"4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338\": container with ID starting with 4e8cbc9a689a75ca8b1039502ad09de39c03fbd9059c20f96cc18707eebfe338 not found: ID does not exist" Jul 09 16:59:22.831943 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.831906 2577 scope.go:117] "RemoveContainer" containerID="886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e" Jul 09 16:59:22.832185 ip-10-0-133-162 kubenswrapper[2577]: E0709 16:59:22.832162 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e\": container with ID starting with 886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e not found: ID does not exist" containerID="886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e" Jul 09 16:59:22.832244 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.832194 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e"} err="failed to get container status \"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e\": rpc error: code = NotFound desc = could not find container \"886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e\": container with ID starting with 886a6654537defeb883e26a0fbef6143bce5d0968b7214d886b14e6cbffbc51e not found: ID does not exist" Jul 09 16:59:22.836625 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.836572 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:59:22.842510 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:22.842487 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-7ab20f-openshift-default-f5d5845dc-2w2hd"] Jul 09 16:59:23.993263 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:23.993223 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" path="/var/lib/kubelet/pods/7595d21b-637c-4ff6-822a-af7c1de09375/volumes" Jul 09 16:59:24.760326 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:24.760293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 16:59:27.787412 ip-10-0-133-162 kubenswrapper[2577]: I0709 16:59:27.787336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:01.433681 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:01.433647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:01.702646 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:01.702549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:01.925238 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:01.925196 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:02.170392 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.170352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:02.508694 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.508643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:02.678152 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.678114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:02.779225 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.779143 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:02.918289 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.918255 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 17:00:02.918546 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:02.918516 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" podUID="4664e641-42bc-42c2-a56b-0ecf320c409a" containerName="maas-api" containerID="cri-o://61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4" gracePeriod=30 Jul 09 17:00:03.173054 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.173024 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 17:00:03.193039 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.192999 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:03.257396 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.257357 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9h45\" (UniqueName: \"kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45\") pod \"4664e641-42bc-42c2-a56b-0ecf320c409a\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " Jul 09 17:00:03.257396 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.257407 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") pod \"4664e641-42bc-42c2-a56b-0ecf320c409a\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " Jul 09 17:00:03.257626 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.257445 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca\") pod \"4664e641-42bc-42c2-a56b-0ecf320c409a\" (UID: \"4664e641-42bc-42c2-a56b-0ecf320c409a\") " Jul 09 17:00:03.257840 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.257816 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4664e641-42bc-42c2-a56b-0ecf320c409a" (UID: "4664e641-42bc-42c2-a56b-0ecf320c409a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:00:03.259665 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.259635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4664e641-42bc-42c2-a56b-0ecf320c409a" (UID: "4664e641-42bc-42c2-a56b-0ecf320c409a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:00:03.259791 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.259691 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45" (OuterVolumeSpecName: "kube-api-access-r9h45") pod "4664e641-42bc-42c2-a56b-0ecf320c409a" (UID: "4664e641-42bc-42c2-a56b-0ecf320c409a"). InnerVolumeSpecName "kube-api-access-r9h45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:03.359276 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.359171 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9h45\" (UniqueName: \"kubernetes.io/projected/4664e641-42bc-42c2-a56b-0ecf320c409a-kube-api-access-r9h45\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:03.359276 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.359220 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4664e641-42bc-42c2-a56b-0ecf320c409a-maas-api-tls\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:03.359276 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:03.359230 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4664e641-42bc-42c2-a56b-0ecf320c409a-openshift-service-ca\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:04.032802 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.032767 2577 generic.go:356] "Generic (PLEG): container finished" podID="4664e641-42bc-42c2-a56b-0ecf320c409a" containerID="61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4" exitCode=0 Jul 09 17:00:04.033291 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.032810 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" Jul 09 17:00:04.033291 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.032859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" event={"ID":"4664e641-42bc-42c2-a56b-0ecf320c409a","Type":"ContainerDied","Data":"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4"} Jul 09 17:00:04.033291 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.032945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg" event={"ID":"4664e641-42bc-42c2-a56b-0ecf320c409a","Type":"ContainerDied","Data":"f93be731161740999415b1da922598ae764285f96670e93729f74d3557df52c8"} Jul 09 17:00:04.033291 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.032985 2577 scope.go:117] "RemoveContainer" containerID="61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4" Jul 09 17:00:04.042551 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.042532 2577 scope.go:117] "RemoveContainer" containerID="61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4" Jul 09 17:00:04.042839 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:00:04.042817 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4\": container with ID starting with 61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4 not found: ID does not exist" containerID="61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4" Jul 09 17:00:04.042904 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.042852 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4"} err="failed to get container status \"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4\": rpc error: code = NotFound desc = could not find container \"61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4\": container with ID starting with 61b7133b1bdbf58473f2280eaf0fd40b5bac0a16fdc8686ec8c0998759b92ef4 not found: ID does not exist" Jul 09 17:00:04.051056 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.051029 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 17:00:04.054434 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:04.054410 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1876d0-7fc6c4485c-c8rrg"] Jul 09 17:00:05.993907 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:05.993875 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4664e641-42bc-42c2-a56b-0ecf320c409a" path="/var/lib/kubelet/pods/4664e641-42bc-42c2-a56b-0ecf320c409a/volumes" Jul 09 17:00:08.512137 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:08.512099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:08.612113 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:08.612073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:08.830893 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:08.830807 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 17:00:08.831079 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:08.831052 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" containerID="cri-o://66c61e97f13ee03c464eb9f8d5856483c68415a3da75d7592d600dc879400201" gracePeriod=30 Jul 09 17:00:08.846540 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:08.846503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:09.040427 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.040387 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:09.388458 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.388425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:09.664978 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.664873 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:09.665481 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665454 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:09.665481 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665481 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665497 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665507 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665518 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665527 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665581 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" containerName="maas-api" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665588 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" containerName="maas-api" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665604 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4664e641-42bc-42c2-a56b-0ecf320c409a" containerName="maas-api" Jul 09 17:00:09.665674 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665609 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4664e641-42bc-42c2-a56b-0ecf320c409a" containerName="maas-api" Jul 09 17:00:09.666087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665692 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:09.666087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665708 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.666087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665715 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8597e24-328c-480a-a0d1-4af7c5883198" containerName="istio-proxy" Jul 09 17:00:09.666087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665722 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2459c6b2-707c-40c6-a1a9-ed59bd3d3742" containerName="maas-api" Jul 09 17:00:09.666087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.665729 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4664e641-42bc-42c2-a56b-0ecf320c409a" containerName="maas-api" Jul 09 17:00:09.670390 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.670367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.673174 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.673153 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0138a7-openshift-default-dockercfg-wktrh\"" Jul 09 17:00:09.680377 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.680353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:09.695221 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.695189 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:09.821259 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821259 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjvz\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821695 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.821695 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.821530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.922752 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.922752 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923049 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.922901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjvz\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923405 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.923340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923405 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.923356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923405 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.923399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923606 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.923575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.923721 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.923700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.925477 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.925452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.925654 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.925633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.931859 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.931831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.932023 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.932004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjvz\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz\") pod \"e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:09.982673 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:09.982633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:10.129403 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:10.129367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:10.131500 ip-10-0-133-162 kubenswrapper[2577]: W0709 17:00:10.131460 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3482047c_4513_4818_9cfe_93181fb32f90.slice/crio-edd48c3c116407be43d8b7cb8639d680f24553524e66cefef32ecfa8085f2dd0 WatchSource:0}: Error finding container edd48c3c116407be43d8b7cb8639d680f24553524e66cefef32ecfa8085f2dd0: Status 404 returned error can't find the container with id edd48c3c116407be43d8b7cb8639d680f24553524e66cefef32ecfa8085f2dd0 Jul 09 17:00:10.133757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:10.133722 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:10.133824 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:10.133797 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:10.133867 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:10.133831 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:10.235371 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:10.235342 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:11.074189 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:11.074148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerStarted","Data":"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b"} Jul 09 17:00:11.074668 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:11.074214 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerStarted","Data":"edd48c3c116407be43d8b7cb8639d680f24553524e66cefef32ecfa8085f2dd0"} Jul 09 17:00:11.096297 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:11.096243 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podStartSLOduration=2.09622707 podStartE2EDuration="2.09622707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:00:11.093715984 +0000 UTC m=+2465.618040319" watchObservedRunningTime="2026-07-09 17:00:11.09622707 +0000 UTC m=+2465.620551402" Jul 09 17:00:11.983711 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:11.983671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:11.993393 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:11.993367 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:12.078889 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:12.078857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:12.079748 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:12.079730 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:13.045531 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:13.045492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:13.128346 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:13.128316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:13.209532 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:13.209502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:13.309173 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:13.309088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:13.730868 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:13.730832 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:14.090481 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.090454 2577 generic.go:356] "Generic (PLEG): container finished" podID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerID="66c61e97f13ee03c464eb9f8d5856483c68415a3da75d7592d600dc879400201" exitCode=0 Jul 09 17:00:14.090639 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.090539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerDied","Data":"66c61e97f13ee03c464eb9f8d5856483c68415a3da75d7592d600dc879400201"} Jul 09 17:00:14.090639 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.090590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" event={"ID":"658dfc2a-adb5-4d91-8528-b917ec64c731","Type":"ContainerDied","Data":"090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b"} Jul 09 17:00:14.090639 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.090607 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090cdf4b1c7694c1d488cbd8d1252df75b1f8d565913d6a3f08182f4804f859b" Jul 09 17:00:14.090639 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.090626 2577 scope.go:117] "RemoveContainer" containerID="371adc5b9a45be4238daea1593d5caf9cf01b1ee474be55b7392eda15f42977a" Jul 09 17:00:14.091430 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.091408 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 17:00:14.192351 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.192307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:14.266250 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266145 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-585z9\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266250 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266205 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266250 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266274 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266320 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266344 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266398 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.266535 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266421 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token\") pod \"658dfc2a-adb5-4d91-8528-b917ec64c731\" (UID: \"658dfc2a-adb5-4d91-8528-b917ec64c731\") " Jul 09 17:00:14.267000 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266572 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:14.267000 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266588 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:14.267000 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266827 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.267000 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266848 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.267000 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.266978 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:14.267404 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.267370 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data" (OuterVolumeSpecName: "istio-data") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:14.267522 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.267457 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:00:14.269246 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.269220 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9" (OuterVolumeSpecName: "kube-api-access-585z9") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "kube-api-access-585z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:14.269372 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.269275 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token" (OuterVolumeSpecName: "istio-token") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:14.269428 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.269413 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:14.269524 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.269504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "658dfc2a-adb5-4d91-8528-b917ec64c731" (UID: "658dfc2a-adb5-4d91-8528-b917ec64c731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 17:00:14.367576 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367546 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367576 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367575 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367576 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367583 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/658dfc2a-adb5-4d91-8528-b917ec64c731-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367811 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367592 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/658dfc2a-adb5-4d91-8528-b917ec64c731-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367811 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367600 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367811 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367612 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:14.367811 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:14.367621 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-585z9\" (UniqueName: \"kubernetes.io/projected/658dfc2a-adb5-4d91-8528-b917ec64c731-kube-api-access-585z9\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:15.097090 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:15.097053 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2" Jul 09 17:00:15.131421 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:15.131387 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 17:00:15.135359 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:15.135330 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1876d0-openshift-default-5bf99ccb4c-ftts2"] Jul 09 17:00:15.994293 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:15.994261 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" path="/var/lib/kubelet/pods/658dfc2a-adb5-4d91-8528-b917ec64c731/volumes" Jul 09 17:00:17.855192 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:17.855135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:18.033783 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.033746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:18.324584 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.324548 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:18.325154 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325133 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325159 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325176 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325185 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325203 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:18.325215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325211 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:18.325490 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325306 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325490 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325317 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="658dfc2a-adb5-4d91-8528-b917ec64c731" containerName="istio-proxy" Jul 09 17:00:18.325490 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.325327 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7595d21b-637c-4ff6-822a-af7c1de09375" containerName="istio-proxy" Jul 09 17:00:18.330223 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.330206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.332992 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.332967 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-254f3b-openshift-default-dockercfg-w8gkq\"" Jul 09 17:00:18.340663 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.340634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:18.354547 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.354496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:18.505581 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.505771 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.505771 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjws\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.505771 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.505771 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.505771 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.506036 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.506036 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.506036 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.505903 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607347 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607347 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607538 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjws\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607538 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607538 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607689 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607689 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607788 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.607847 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.607795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.608050 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.608023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.608140 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.608077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.608306 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.608286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.608365 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.608288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.608445 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.608417 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.609986 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.609910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.610236 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.610217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.611534 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.611489 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:18.615424 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.615396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.615705 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.615689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjws\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws\") pod \"e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.642182 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.642154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:18.788051 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.788026 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:18.789932 ip-10-0-133-162 kubenswrapper[2577]: W0709 17:00:18.789885 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a0db6d_b615_48ae_a4f6_82f6852d4647.slice/crio-fd54581da60b18bc8e65ac040122be3a073cfb562f7987b51a4fea20d07721a8 WatchSource:0}: Error finding container fd54581da60b18bc8e65ac040122be3a073cfb562f7987b51a4fea20d07721a8: Status 404 returned error can't find the container with id fd54581da60b18bc8e65ac040122be3a073cfb562f7987b51a4fea20d07721a8 Jul 09 17:00:18.791981 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.791945 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:18.792086 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.792019 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:18.792086 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:18.792053 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:19.119556 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:19.119522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerStarted","Data":"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289"} Jul 09 17:00:19.119965 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:19.119571 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerStarted","Data":"fd54581da60b18bc8e65ac040122be3a073cfb562f7987b51a4fea20d07721a8"} Jul 09 17:00:19.140840 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:19.140788 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podStartSLOduration=1.140772694 podStartE2EDuration="1.140772694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:00:19.138497894 +0000 UTC m=+2473.662822229" watchObservedRunningTime="2026-07-09 17:00:19.140772694 +0000 UTC m=+2473.665097026" Jul 09 17:00:19.642443 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:19.642406 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:19.647466 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:19.647439 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:20.125683 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:20.125643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:20.126647 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:20.126627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:21.141102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:21.141071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:21.797703 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:21.797667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:21.897681 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:21.897646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:22.034698 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:22.034666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:22.187677 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:22.187586 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:25.601328 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:25.601269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:31.183860 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.183822 2577 generic.go:356] "Generic (PLEG): container finished" podID="3482047c-4513-4818-9cfe-93181fb32f90" containerID="e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b" exitCode=0 Jul 09 17:00:31.184323 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.183892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerDied","Data":"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b"} Jul 09 17:00:31.184386 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.184327 2577 scope.go:117] "RemoveContainer" containerID="e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b" Jul 09 17:00:31.185413 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.185377 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:31.185531 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.185478 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:31.185603 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:31.185529 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:32.193809 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:32.193771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerStarted","Data":"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8"} Jul 09 17:00:32.984007 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:32.983966 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:32.984210 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:32.984055 2577 patch_prober.go:28] interesting pod/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 17:00:32.984210 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:32.984104 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 17:00:33.809048 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:33.809012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:33.984114 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:33.984061 2577 patch_prober.go:28] interesting pod/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 17:00:33.984306 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:33.984147 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 17:00:34.164378 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:34.164295 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:34.318578 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:34.318464 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:34.525649 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:34.525529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:34.983506 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:34.983468 2577 patch_prober.go:28] interesting pod/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 17:00:34.983876 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:34.983531 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 17:00:35.754329 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:35.754298 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:35.984002 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:35.983955 2577 patch_prober.go:28] interesting pod/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 17:00:35.984464 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:35.984033 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 17:00:36.167217 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:36.167128 2577 patch_prober.go:28] interesting pod/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 17:00:36.167368 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:36.167207 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 17:00:36.986840 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:36.986806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:36.987373 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:36.987350 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:36.988012 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:36.987993 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:38.245230 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.245195 2577 generic.go:356] "Generic (PLEG): container finished" podID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerID="3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289" exitCode=0 Jul 09 17:00:38.245609 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.245263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerDied","Data":"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289"} Jul 09 17:00:38.245749 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.245731 2577 scope.go:117] "RemoveContainer" containerID="3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289" Jul 09 17:00:38.246774 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.246744 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:38.246827 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.246812 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:38.246861 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:38.246852 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:39.253542 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.253505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerStarted","Data":"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d"} Jul 09 17:00:39.643136 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.643041 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:39.643292 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.643269 2577 patch_prober.go:28] interesting pod/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 17:00:39.643339 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.643323 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 17:00:39.820386 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.820347 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:39.921282 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.921190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:39.982164 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.982126 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:39.982335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:39.982312 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" containerID="cri-o://e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8" gracePeriod=30 Jul 09 17:00:40.643418 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:40.643379 2577 patch_prober.go:28] interesting pod/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 17:00:40.643853 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:40.643450 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 17:00:41.160179 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:41.160147 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:41.643275 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:41.643239 2577 patch_prober.go:28] interesting pod/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 17:00:41.643462 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:41.643299 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 17:00:42.483123 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:42.483082 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:42.643069 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:42.643033 2577 patch_prober.go:28] interesting pod/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 09 17:00:42.643242 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:42.643092 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 09 17:00:43.647025 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:43.646984 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:43.647426 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:43.647281 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:43.648098 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:43.648074 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:45.236253 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.236188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:45.288502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.288416 2577 generic.go:356] "Generic (PLEG): container finished" podID="3482047c-4513-4818-9cfe-93181fb32f90" containerID="e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8" exitCode=0 Jul 09 17:00:45.288502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.288462 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" Jul 09 17:00:45.288677 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.288503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerDied","Data":"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8"} Jul 09 17:00:45.288677 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.288556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f" event={"ID":"3482047c-4513-4818-9cfe-93181fb32f90","Type":"ContainerDied","Data":"edd48c3c116407be43d8b7cb8639d680f24553524e66cefef32ecfa8085f2dd0"} Jul 09 17:00:45.288677 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.288580 2577 scope.go:117] "RemoveContainer" containerID="e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8" Jul 09 17:00:45.298576 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.298558 2577 scope.go:117] "RemoveContainer" containerID="e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b" Jul 09 17:00:45.307083 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.307059 2577 scope.go:117] "RemoveContainer" containerID="e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8" Jul 09 17:00:45.307379 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:00:45.307357 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8\": container with ID starting with e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8 not found: ID does not exist" containerID="e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8" Jul 09 17:00:45.307443 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.307390 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8"} err="failed to get container status \"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8\": rpc error: code = NotFound desc = could not find container \"e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8\": container with ID starting with e3caa4016b91a280791cc299ca74246cdc45dd4ce4bf614b2e568372fd4bbbb8 not found: ID does not exist" Jul 09 17:00:45.307443 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.307409 2577 scope.go:117] "RemoveContainer" containerID="e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b" Jul 09 17:00:45.307672 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:00:45.307654 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b\": container with ID starting with e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b not found: ID does not exist" containerID="e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b" Jul 09 17:00:45.307732 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.307682 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b"} err="failed to get container status \"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b\": rpc error: code = NotFound desc = could not find container \"e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b\": container with ID starting with e29da41ca0b523c50611456ef1979ea60ea05d9009a5fc68f831f99b66a7d44b not found: ID does not exist" Jul 09 17:00:45.361969 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.361908 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.361969 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.361967 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362046 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjvz\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362067 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362134 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362215 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362167 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362218 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo\") pod \"3482047c-4513-4818-9cfe-93181fb32f90\" (UID: \"3482047c-4513-4818-9cfe-93181fb32f90\") " Jul 09 17:00:45.362502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362244 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:45.362502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362359 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:45.362502 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362378 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:45.362719 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362617 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.362719 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362636 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.362719 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362649 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.362719 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.362675 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:00:45.363586 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.363556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data" (OuterVolumeSpecName: "istio-data") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:45.364489 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.364459 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:45.364574 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.364504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz" (OuterVolumeSpecName: "kube-api-access-fkjvz") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "kube-api-access-fkjvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:45.364868 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.364849 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 17:00:45.364960 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.364862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token" (OuterVolumeSpecName: "istio-token") pod "3482047c-4513-4818-9cfe-93181fb32f90" (UID: "3482047c-4513-4818-9cfe-93181fb32f90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:45.463306 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463263 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3482047c-4513-4818-9cfe-93181fb32f90-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.463306 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463301 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.463306 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463312 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3482047c-4513-4818-9cfe-93181fb32f90-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.463551 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463323 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3482047c-4513-4818-9cfe-93181fb32f90-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.463551 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463335 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjvz\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-kube-api-access-fkjvz\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.463551 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.463347 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3482047c-4513-4818-9cfe-93181fb32f90-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:45.610682 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.610653 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:45.614721 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.614691 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0138a7-openshift-default-6b979c79f9-zhv9f"] Jul 09 17:00:45.993175 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:45.993145 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3482047c-4513-4818-9cfe-93181fb32f90" path="/var/lib/kubelet/pods/3482047c-4513-4818-9cfe-93181fb32f90/volumes" Jul 09 17:00:46.695202 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:46.695170 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:46.797900 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:46.797857 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:46.984975 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:46.984939 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:46.985196 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:46.985171 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" containerID="cri-o://b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d" gracePeriod=30 Jul 09 17:00:47.014898 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.014861 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:47.195250 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.195211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:47.537074 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.537036 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:47.804269 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804189 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:00:47.804605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804564 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.804605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804575 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.804605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804589 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.804605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804594 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.804735 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804659 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.804735 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.804668 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3482047c-4513-4818-9cfe-93181fb32f90" containerName="istio-proxy" Jul 09 17:00:47.809286 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.809265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.811779 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.811757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d1f7bd-openshift-default-dockercfg-zhp7r\"" Jul 09 17:00:47.820641 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.820611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:00:47.834971 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.834941 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:47.886875 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.886833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887089 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.886877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887089 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.886972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887089 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887089 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887245 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887245 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887322 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfxp\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.887322 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.887298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.987766 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.987724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.987986 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.987783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.987986 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.987801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.987986 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.987945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988163 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988431 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988568 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988639 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988700 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988758 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988814 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.988814 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.988803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfxp\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.989213 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.989190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.990625 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.990599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:47.990829 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:47.990651 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:48.000663 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.000634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfxp\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:48.001060 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.001036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token\") pod \"e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:48.122290 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.122194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:48.276189 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.276161 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:00:48.278351 ip-10-0-133-162 kubenswrapper[2577]: W0709 17:00:48.278315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode939e1a5_917c_4688_83d3_589eaa03d20a.slice/crio-6fe51dc8a964015366d6647de697c99ebe8d748e1ae59e50e3fabe62e9756cf9 WatchSource:0}: Error finding container 6fe51dc8a964015366d6647de697c99ebe8d748e1ae59e50e3fabe62e9756cf9: Status 404 returned error can't find the container with id 6fe51dc8a964015366d6647de697c99ebe8d748e1ae59e50e3fabe62e9756cf9 Jul 09 17:00:48.283102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.283069 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:48.283191 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.283141 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:48.283191 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.283169 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:48.305696 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.305669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerStarted","Data":"6fe51dc8a964015366d6647de697c99ebe8d748e1ae59e50e3fabe62e9756cf9"} Jul 09 17:00:48.309179 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:48.309153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:49.312577 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:49.312536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerStarted","Data":"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3"} Jul 09 17:00:49.337962 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:49.337881 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podStartSLOduration=2.3378651440000002 podStartE2EDuration="2.337865144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:00:49.335552358 +0000 UTC m=+2503.859876691" watchObservedRunningTime="2026-07-09 17:00:49.337865144 +0000 UTC m=+2503.862189525" Jul 09 17:00:50.122383 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:50.122347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:50.127574 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:50.127546 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:50.317668 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:50.317635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:50.318812 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:50.318791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:00:51.331612 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:51.331576 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:51.446543 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:51.446506 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:51.564950 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:51.564897 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:51.820778 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:51.820743 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:52.069746 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.069709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:52.166466 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.166432 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:52.244469 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.244444 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:52.330222 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.330182 2577 generic.go:356] "Generic (PLEG): container finished" podID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerID="b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d" exitCode=0 Jul 09 17:00:52.330406 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.330237 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" Jul 09 17:00:52.330406 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.330277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerDied","Data":"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d"} Jul 09 17:00:52.330406 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.330345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5" event={"ID":"b1a0db6d-b615-48ae-a4f6-82f6852d4647","Type":"ContainerDied","Data":"fd54581da60b18bc8e65ac040122be3a073cfb562f7987b51a4fea20d07721a8"} Jul 09 17:00:52.330406 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.330379 2577 scope.go:117] "RemoveContainer" containerID="b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d" Jul 09 17:00:52.332773 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332748 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bjws\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332816 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332844 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332874 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332900 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332947 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.332974 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333012 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert\") pod \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\" (UID: \"b1a0db6d-b615-48ae-a4f6-82f6852d4647\") " Jul 09 17:00:52.333237 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333154 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:52.333664 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333354 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.334046 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333972 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:52.334046 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.333993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:52.334208 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.334063 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:00:52.335457 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.335430 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws" (OuterVolumeSpecName: "kube-api-access-8bjws") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "kube-api-access-8bjws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:52.335970 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.335942 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:52.336087 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.336014 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:00:52.336537 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.336512 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:00:52.337228 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.337207 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1a0db6d-b615-48ae-a4f6-82f6852d4647" (UID: "b1a0db6d-b615-48ae-a4f6-82f6852d4647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 17:00:52.355251 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.355228 2577 scope.go:117] "RemoveContainer" containerID="3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289" Jul 09 17:00:52.363460 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.363438 2577 scope.go:117] "RemoveContainer" containerID="b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d" Jul 09 17:00:52.363744 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:00:52.363725 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d\": container with ID starting with b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d not found: ID does not exist" containerID="b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d" Jul 09 17:00:52.363790 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.363752 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d"} err="failed to get container status \"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d\": rpc error: code = NotFound desc = could not find container \"b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d\": container with ID starting with b8b3be5ab8ed4ff0ddbcd12f2db64afb0340958ba17ffff84a85a2b82d8b452d not found: ID does not exist" Jul 09 17:00:52.363790 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.363774 2577 scope.go:117] "RemoveContainer" containerID="3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289" Jul 09 17:00:52.364074 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:00:52.364053 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289\": container with ID starting with 3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289 not found: ID does not exist" containerID="3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289" Jul 09 17:00:52.364136 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.364079 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289"} err="failed to get container status \"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289\": rpc error: code = NotFound desc = could not find container \"3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289\": container with ID starting with 3a8a09f0e76f5858bfea6ea3b59de417014fc4e43c30bfbe3a93e8d253768289 not found: ID does not exist" Jul 09 17:00:52.434112 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434069 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjws\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-kube-api-access-8bjws\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434112 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434103 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434112 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434118 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434355 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434130 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434355 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434142 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434355 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434154 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434355 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434165 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.434355 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.434176 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1a0db6d-b615-48ae-a4f6-82f6852d4647-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:00:52.652808 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.652772 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:52.656795 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:52.656765 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-254f3b-openshift-default-59f7fb9745-fn5p5"] Jul 09 17:00:53.993182 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:53.993143 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" path="/var/lib/kubelet/pods/b1a0db6d-b615-48ae-a4f6-82f6852d4647/volumes" Jul 09 17:00:55.805488 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:55.805450 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:56.154605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.154517 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:56.448903 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.448817 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:00:56.449253 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449239 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.449350 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449255 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.449350 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449265 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.449350 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449270 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.449350 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449342 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.449492 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.449353 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a0db6d-b615-48ae-a4f6-82f6852d4647" containerName="istio-proxy" Jul 09 17:00:56.453375 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.453348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.456138 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.456115 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e779af-openshift-default-dockercfg-755cq\"" Jul 09 17:00:56.464509 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.464481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:00:56.515808 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.515777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:56.573201 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573372 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573372 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573372 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573475 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573475 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573475 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573570 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5ql\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.573570 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.573515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674459 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674459 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674459 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5ql\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.674757 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.675127 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.674827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.675190 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.675130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.675370 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.675347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.675539 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.675394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.675620 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.675566 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.677055 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.677031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.677134 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.677102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.685538 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.685510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.685686 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.685655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5ql\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql\") pod \"e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.766136 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.766092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:56.829813 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.829779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:56.915601 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.915572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:00:56.918492 ip-10-0-133-162 kubenswrapper[2577]: W0709 17:00:56.918462 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f439e2c_3f21_4aa0_a063_d3bfaf059f14.slice/crio-66fc147733cabe9aa8213496f861395bde3d94d2a9e48dec1880f382f8e8aaff WatchSource:0}: Error finding container 66fc147733cabe9aa8213496f861395bde3d94d2a9e48dec1880f382f8e8aaff: Status 404 returned error can't find the container with id 66fc147733cabe9aa8213496f861395bde3d94d2a9e48dec1880f382f8e8aaff Jul 09 17:00:56.920561 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.920528 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:56.920649 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.920594 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:56.920649 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:56.920624 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:00:57.357524 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:57.357488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerStarted","Data":"9558c4c9270f3dfe00cf38d39f3d645ae89fae14ccf2aee53666d06c691f7502"} Jul 09 17:00:57.357722 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:57.357553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerStarted","Data":"66fc147733cabe9aa8213496f861395bde3d94d2a9e48dec1880f382f8e8aaff"} Jul 09 17:00:57.379302 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:57.379250 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podStartSLOduration=1.379234221 podStartE2EDuration="1.379234221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:00:57.377600096 +0000 UTC m=+2511.901924428" watchObservedRunningTime="2026-07-09 17:00:57.379234221 +0000 UTC m=+2511.903558552" Jul 09 17:00:57.767230 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:57.767192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:57.772209 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:57.772179 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:58.362420 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:58.362381 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:58.363501 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:58.363481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:00:59.499196 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:59.499153 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:00:59.593937 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:00:59.593880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:00.060293 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:00.060236 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:00.160341 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:00.160310 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:00.267739 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:00.267707 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:03.970532 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:03.970500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:04.689391 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:04.689355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:05.319448 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.319363 2577 patch_prober.go:28] interesting pod/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 17:01:05.319448 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.319422 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 17:01:05.401848 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.401765 2577 generic.go:356] "Generic (PLEG): container finished" podID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerID="017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3" exitCode=0 Jul 09 17:01:05.402007 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.401844 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerDied","Data":"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3"} Jul 09 17:01:05.402285 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.402271 2577 scope.go:117] "RemoveContainer" containerID="017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3" Jul 09 17:01:05.403363 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.403335 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:05.403429 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.403408 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:05.403464 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:05.403439 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:06.439043 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:06.438422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerStarted","Data":"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879"} Jul 09 17:01:07.122853 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:07.122792 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:01:07.123101 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:07.122852 2577 patch_prober.go:28] interesting pod/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 17:01:07.123101 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:07.122908 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 17:01:08.122889 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:08.122851 2577 patch_prober.go:28] interesting pod/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 17:01:08.123505 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:08.123468 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 17:01:09.123473 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:09.123422 2577 patch_prober.go:28] interesting pod/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 17:01:09.124008 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:09.123517 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 17:01:10.122721 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:10.122668 2577 patch_prober.go:28] interesting pod/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 17:01:10.122981 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:10.122744 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 17:01:11.127442 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:11.127407 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:01:11.127968 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:11.127948 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:01:11.128968 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:11.128947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:01:12.929098 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:12.929063 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:13.029680 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.029638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:13.397035 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.396188 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:46190->10.132.0.60:15021: read: connection reset by peer" start-of-body= Jul 09 17:01:13.397035 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.396286 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:46190->10.132.0.60:15021: read: connection reset by peer" Jul 09 17:01:13.477273 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.477240 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerID="9558c4c9270f3dfe00cf38d39f3d645ae89fae14ccf2aee53666d06c691f7502" exitCode=0 Jul 09 17:01:13.477436 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.477309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerDied","Data":"9558c4c9270f3dfe00cf38d39f3d645ae89fae14ccf2aee53666d06c691f7502"} Jul 09 17:01:13.477737 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.477723 2577 scope.go:117] "RemoveContainer" containerID="9558c4c9270f3dfe00cf38d39f3d645ae89fae14ccf2aee53666d06c691f7502" Jul 09 17:01:13.478853 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.478819 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:13.478938 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.478897 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:13.478997 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:13.478958 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 09 17:01:14.484045 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:14.484007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerStarted","Data":"085a86c351f6b04176538a356a29aea3f8b6e7d24c56dcbbb7835926421983a3"} Jul 09 17:01:14.767063 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:14.766960 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:01:14.767332 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:14.767307 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 09 17:01:14.767388 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:14.767366 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 09 17:01:15.766805 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:15.766757 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 09 17:01:15.767251 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:15.766838 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 09 17:01:15.869459 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:15.869425 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:16.616243 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:16.616202 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:16.767070 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:16.767028 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 09 17:01:16.767551 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:16.767107 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 09 17:01:17.767541 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:17.767496 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 09 17:01:17.768066 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:17.767611 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 09 17:01:18.766722 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:18.766683 2577 patch_prober.go:28] interesting pod/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 09 17:01:18.766881 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:18.766751 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 09 17:01:19.770795 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:19.770763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:01:19.771266 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:19.771149 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:01:19.771848 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:19.771823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:01:20.874938 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:20.874895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:20.973660 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:20.973619 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:21.849879 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:21.849831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:43.178394 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:43.178357 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:01:59.889652 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:01:59.889609 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:02:06.434114 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:02:06.434080 2577 scope.go:117] "RemoveContainer" containerID="66c61e97f13ee03c464eb9f8d5856483c68415a3da75d7592d600dc879400201" Jul 09 17:02:46.339859 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:02:46.339783 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:03:54.831227 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:03:54.831181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:03:54.932224 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:03:54.932190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:04:06.128130 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:06.128096 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 17:04:06.136605 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:06.136573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 17:04:06.140293 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:06.140268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 17:04:06.146696 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:06.146667 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8482k_b74aa39e-e278-4643-b285-59f08edc7821/ovn-acl-logging/0.log" Jul 09 17:04:06.357248 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:06.357211 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:04:07.352118 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:07.352086 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:04:11.300024 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:11.299988 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:04:12.662349 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:12.662315 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:04:41.858431 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:04:41.858399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:05:26.452699 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:05:26.452654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:05:27.337104 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:05:27.337062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:05:56.462222 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:05:56.462182 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:13.705470 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:13.705422 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:13.806762 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:13.806724 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:15.947495 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:15.947444 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:21.645493 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:21.645453 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:21.746900 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:21.746854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:30.353375 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:30.353333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:44.526104 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:44.526062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:58.754696 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:58.754611 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:06:59.629112 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:06:59.629071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:41.497244 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:41.497210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:41.865248 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:41.865145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:47.159197 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.159159 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:47.260211 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.260174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:47.410085 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.409994 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:07:47.410361 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.410216 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" containerID="cri-o://5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879" gracePeriod=30 Jul 09 17:07:47.425398 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.425367 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:47.671934 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.671834 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:47.932160 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:47.932084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:48.296094 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:48.296057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:48.492991 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:48.492955 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:48.595635 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:48.595532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:48.765748 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:48.765713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:52.670956 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.670873 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:07:52.693159 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693129 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.693335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693195 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.693335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693227 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.693335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693249 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.693335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693271 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.693335 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693376 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693466 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.693627 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctfxp\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp\") pod \"e939e1a5-917c-4688-83d3-589eaa03d20a\" (UID: \"e939e1a5-917c-4688-83d3-589eaa03d20a\") " Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.694270 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.695009 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.696208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:07:52.696271 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.696218 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:07:52.696627 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.696290 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:52.696627 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.694730 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:52.696627 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.696476 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 17:07:52.696808 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.696785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:52.697079 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.697054 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp" (OuterVolumeSpecName: "kube-api-access-ctfxp") pod "e939e1a5-917c-4688-83d3-589eaa03d20a" (UID: "e939e1a5-917c-4688-83d3-589eaa03d20a"). InnerVolumeSpecName "kube-api-access-ctfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:07:52.794791 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794755 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.794791 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794784 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.794791 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794794 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e939e1a5-917c-4688-83d3-589eaa03d20a-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.794791 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794802 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.795093 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794809 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.795093 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794818 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.795093 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794827 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.795093 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794836 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e939e1a5-917c-4688-83d3-589eaa03d20a-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:52.795093 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:52.794845 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfxp\" (UniqueName: \"kubernetes.io/projected/e939e1a5-917c-4688-83d3-589eaa03d20a-kube-api-access-ctfxp\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:53.505890 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.505850 2577 generic.go:356] "Generic (PLEG): container finished" podID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerID="5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879" exitCode=0 Jul 09 17:07:53.506102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.505888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerDied","Data":"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879"} Jul 09 17:07:53.506102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.505901 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" Jul 09 17:07:53.506102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.505975 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7" event={"ID":"e939e1a5-917c-4688-83d3-589eaa03d20a","Type":"ContainerDied","Data":"6fe51dc8a964015366d6647de697c99ebe8d748e1ae59e50e3fabe62e9756cf9"} Jul 09 17:07:53.506102 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.506001 2577 scope.go:117] "RemoveContainer" containerID="5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879" Jul 09 17:07:53.515738 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.515723 2577 scope.go:117] "RemoveContainer" containerID="017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3" Jul 09 17:07:53.526135 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.526110 2577 scope.go:117] "RemoveContainer" containerID="5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879" Jul 09 17:07:53.526448 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:07:53.526425 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879\": container with ID starting with 5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879 not found: ID does not exist" containerID="5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879" Jul 09 17:07:53.526563 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.526461 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879"} err="failed to get container status \"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879\": rpc error: code = NotFound desc = could not find container \"5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879\": container with ID starting with 5cefc2e79309ed4fc502f8ccbac35d6cb7e96a2930250d7998ca47cb95722879 not found: ID does not exist" Jul 09 17:07:53.526563 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.526486 2577 scope.go:117] "RemoveContainer" containerID="017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3" Jul 09 17:07:53.526786 ip-10-0-133-162 kubenswrapper[2577]: E0709 17:07:53.526765 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3\": container with ID starting with 017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3 not found: ID does not exist" containerID="017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3" Jul 09 17:07:53.526826 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.526798 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3"} err="failed to get container status \"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3\": rpc error: code = NotFound desc = could not find container \"017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3\": container with ID starting with 017c3c510f57242ed99c997556589d54d159968e539cf9cd8a280296a6e7f2d3 not found: ID does not exist" Jul 09 17:07:53.532506 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.532479 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:07:53.537062 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.537038 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d1f7bd-openshift-default-6c766889f9-fhmx7"] Jul 09 17:07:53.993883 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:53.993847 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" path="/var/lib/kubelet/pods/e939e1a5-917c-4688-83d3-589eaa03d20a/volumes" Jul 09 17:07:54.195512 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.195476 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:54.295456 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.295353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:54.411135 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.411099 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:07:54.411455 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.411422 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" containerID="cri-o://085a86c351f6b04176538a356a29aea3f8b6e7d24c56dcbbb7835926421983a3" gracePeriod=30 Jul 09 17:07:54.435987 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.435949 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:54.652819 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:54.652727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:07:59.537160 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.537132 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerID="085a86c351f6b04176538a356a29aea3f8b6e7d24c56dcbbb7835926421983a3" exitCode=0 Jul 09 17:07:59.537495 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.537183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerDied","Data":"085a86c351f6b04176538a356a29aea3f8b6e7d24c56dcbbb7835926421983a3"} Jul 09 17:07:59.537495 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.537236 2577 scope.go:117] "RemoveContainer" containerID="9558c4c9270f3dfe00cf38d39f3d645ae89fae14ccf2aee53666d06c691f7502" Jul 09 17:07:59.681937 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.681856 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:07:59.763993 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.763958 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764002 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764035 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764080 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc5ql\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764183 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764178 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764503 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764208 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764503 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764272 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert\") pod \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\" (UID: \"6f439e2c-3f21-4aa0-a063-d3bfaf059f14\") " Jul 09 17:07:59.764503 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764310 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:59.764503 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764452 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:59.764724 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764582 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-certs\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.764724 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764599 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:59.764724 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764605 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-workload-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.765086 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.764762 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:07:59.765224 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.765199 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:59.766501 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.766476 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql" (OuterVolumeSpecName: "kube-api-access-zc5ql") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "kube-api-access-zc5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:07:59.766727 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.766695 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:07:59.767002 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.766977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 17:07:59.767002 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.766979 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f439e2c-3f21-4aa0-a063-d3bfaf059f14" (UID: "6f439e2c-3f21-4aa0-a063-d3bfaf059f14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:07:59.866147 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866103 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-token\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866147 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866138 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5ql\" (UniqueName: \"kubernetes.io/projected/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-kube-api-access-zc5ql\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866147 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866152 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-credential-socket\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866414 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866163 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-podinfo\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866414 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866175 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-data\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866414 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866188 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istiod-ca-cert\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:07:59.866414 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:07:59.866198 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f439e2c-3f21-4aa0-a063-d3bfaf059f14-istio-envoy\") on node \"ip-10-0-133-162.ec2.internal\" DevicePath \"\"" Jul 09 17:08:00.163594 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.163563 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bc6967554-zdbsq_13464a1c-7fbf-47f4-84af-ef7cabc3b2da/manager/0.log" Jul 09 17:08:00.549192 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.549148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" event={"ID":"6f439e2c-3f21-4aa0-a063-d3bfaf059f14","Type":"ContainerDied","Data":"66fc147733cabe9aa8213496f861395bde3d94d2a9e48dec1880f382f8e8aaff"} Jul 09 17:08:00.549655 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.549197 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w" Jul 09 17:08:00.549655 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.549247 2577 scope.go:117] "RemoveContainer" containerID="085a86c351f6b04176538a356a29aea3f8b6e7d24c56dcbbb7835926421983a3" Jul 09 17:08:00.568079 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.568055 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:08:00.572671 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.572636 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e779af-openshift-default-5fdb7b96c-d2q2w"] Jul 09 17:08:00.772013 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.771987 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd696597f-vp54j_18b8ed4e-e86e-466a-b06e-46b52cb3fe92/copy-manifests/0.log" Jul 09 17:08:00.778798 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:00.778761 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd696597f-vp54j_18b8ed4e-e86e-466a-b06e-46b52cb3fe92/manager/0.log" Jul 09 17:08:01.126635 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:01.126602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bcf4b654b-8zlrf_25ea7695-4b83-4f9e-885d-9f4c04e212ed/maas-api/0.log" Jul 09 17:08:01.242038 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:01.242008 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-m9mq9_8052c29b-8b80-4a88-afe0-17fed5654f4a/postgres/0.log" Jul 09 17:08:01.993130 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:01.993097 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" path="/var/lib/kubelet/pods/6f439e2c-3f21-4aa0-a063-d3bfaf059f14/volumes" Jul 09 17:08:02.032089 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:02.032062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:08:02.651752 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:02.651724 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-x5tpf_5e035f87-0c3b-41b2-8cc0-ae55c2f14c08/manager/0.log" Jul 09 17:08:02.766452 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:02.766423 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xbdht_877460aa-b6be-4672-bb93-5a296a103fee/manager/0.log" Jul 09 17:08:02.981092 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:02.981032 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-n6svw_ae95a51b-6238-4d90-8e10-02741e66c86c/registry-server/0.log" Jul 09 17:08:04.373647 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:04.373620 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-68l6r_172d296a-79ab-4da7-bf3e-f6618d38a54c/payload-processing/0.log" Jul 09 17:08:04.486728 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:04.486681 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6db5c95b-l6zp4_5eb33ab3-d8d8-438c-87c5-d70acafa0db7/router/0.log" Jul 09 17:08:04.824118 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:04.824087 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z9jk6_a28d7bf5-9c25-4e5b-bb90-f05a0668cfc1/main/0.log" Jul 09 17:08:05.049177 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:05.049141 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8nxwd_eeafa3b1-667e-457e-a92c-72c900810ef7/main/0.log" Jul 09 17:08:09.773330 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:09.773292 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:08:09.832370 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:09.832336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" limit=1024 Jul 09 17:08:12.549481 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:12.549432 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kr65g_19baab78-2289-4c52-a951-febcda3d048a/global-pull-secret-syncer/0.log" Jul 09 17:08:12.647448 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:12.647417 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jcrhw_0b8c7abf-3301-4730-a0f4-35d9c5bffe69/konnectivity-agent/0.log" Jul 09 17:08:12.739029 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:12.738993 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-162.ec2.internal_d48bc14c4e35da0c778b71f5201769c5/haproxy/0.log" Jul 09 17:08:18.402633 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:18.402592 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-x5tpf_5e035f87-0c3b-41b2-8cc0-ae55c2f14c08/manager/0.log" Jul 09 17:08:18.429933 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:18.429898 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xbdht_877460aa-b6be-4672-bb93-5a296a103fee/manager/0.log" Jul 09 17:08:18.499819 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:18.499772 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-n6svw_ae95a51b-6238-4d90-8e10-02741e66c86c/registry-server/0.log" Jul 09 17:08:20.620604 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:20.620567 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-nlql6_661c473a-c88b-430d-b27f-191bcb18a652/cluster-monitoring-operator/0.log" Jul 09 17:08:20.822194 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:20.822162 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6857q_681bd3f6-5d93-46b8-8f7f-f8c906212618/node-exporter/0.log" Jul 09 17:08:20.849813 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:20.849785 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6857q_681bd3f6-5d93-46b8-8f7f-f8c906212618/kube-rbac-proxy/0.log" Jul 09 17:08:20.874570 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:20.874501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6857q_681bd3f6-5d93-46b8-8f7f-f8c906212618/init-textfile/0.log" Jul 09 17:08:21.390359 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:21.390329 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-dpmww_a135200a-fc55-4e49-beb5-61405c9132b4/prometheus-operator-admission-webhook/0.log" Jul 09 17:08:22.754770 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:22.754740 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-7nvdj_9c3a8f6e-1b40-4d45-894f-3d7e6bed442d/networking-console-plugin/0.log" Jul 09 17:08:23.344158 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.344127 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/1.log" Jul 09 17:08:23.348376 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.348353 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-6b2fz_6a3a571e-9216-47f1-8c0a-22afb3279d31/console-operator/2.log" Jul 09 17:08:23.386692 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.386660 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j"] Jul 09 17:08:23.387079 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387065 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" Jul 09 17:08:23.387132 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" Jul 09 17:08:23.387132 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387093 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" Jul 09 17:08:23.387132 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387099 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" Jul 09 17:08:23.387132 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387111 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" Jul 09 17:08:23.387132 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387116 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" Jul 09 17:08:23.387290 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387182 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" Jul 09 17:08:23.387290 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387190 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f439e2c-3f21-4aa0-a063-d3bfaf059f14" containerName="istio-proxy" Jul 09 17:08:23.387290 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.387196 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e939e1a5-917c-4688-83d3-589eaa03d20a" containerName="istio-proxy" Jul 09 17:08:23.390505 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.390484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.393086 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.393064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jqtn\"/\"kube-root-ca.crt\"" Jul 09 17:08:23.393194 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.393068 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jqtn\"/\"openshift-service-ca.crt\"" Jul 09 17:08:23.394405 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.394383 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7jqtn\"/\"default-dockercfg-r67xr\"" Jul 09 17:08:23.400217 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.400194 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j"] Jul 09 17:08:23.488948 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.488899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-proc\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.489133 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.489005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-podres\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.489133 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.489043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-lib-modules\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.489133 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.489097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lhb\" (UniqueName: \"kubernetes.io/projected/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-kube-api-access-t9lhb\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.489133 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.489131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-sys\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.589971 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.589900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lhb\" (UniqueName: \"kubernetes.io/projected/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-kube-api-access-t9lhb\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.589987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-sys\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-proc\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-podres\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-lib-modules\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-sys\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590169 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-proc\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590385 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-lib-modules\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.590385 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.590247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-podres\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.598607 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.598541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lhb\" (UniqueName: \"kubernetes.io/projected/cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11-kube-api-access-t9lhb\") pod \"perf-node-gather-daemonset-4jc6j\" (UID: \"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11\") " pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.701160 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.701119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:23.849573 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.843002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j"] Jul 09 17:08:23.850734 ip-10-0-133-162 kubenswrapper[2577]: W0709 17:08:23.850702 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcc7e5166_23ff_4fee_bc3e_d9e7d70c3c11.slice/crio-14894c4f9e0447a5043973ee85d1c5efac93fc566c9de95f6c40cd10bdd85c10 WatchSource:0}: Error finding container 14894c4f9e0447a5043973ee85d1c5efac93fc566c9de95f6c40cd10bdd85c10: Status 404 returned error can't find the container with id 14894c4f9e0447a5043973ee85d1c5efac93fc566c9de95f6c40cd10bdd85c10 Jul 09 17:08:23.892447 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:23.892417 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-h8jff_4c3874e5-08a1-4b77-a197-1ab2a2f8d483/download-server/0.log" Jul 09 17:08:24.471211 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:24.471179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-qdqml_fce00d72-29d4-4fc0-8ddd-8b20f4a39a39/volume-data-source-validator/0.log" Jul 09 17:08:24.666937 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:24.666882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" event={"ID":"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11","Type":"ContainerStarted","Data":"8aa0111d573d5f266899f304d800bedf851afe717cc1d24c9ba006f109e841c4"} Jul 09 17:08:24.667126 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:24.666958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" event={"ID":"cc7e5166-23ff-4fee-bc3e-d9e7d70c3c11","Type":"ContainerStarted","Data":"14894c4f9e0447a5043973ee85d1c5efac93fc566c9de95f6c40cd10bdd85c10"} Jul 09 17:08:24.667126 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:24.666981 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:24.684204 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:24.684150 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" podStartSLOduration=1.684133469 podStartE2EDuration="1.684133469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:08:24.683796593 +0000 UTC m=+2959.208120927" watchObservedRunningTime="2026-07-09 17:08:24.684133469 +0000 UTC m=+2959.208457801" Jul 09 17:08:25.581104 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:25.581077 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zfd8z_9efaa331-652f-44ca-9f42-ba4a536de051/dns/0.log" Jul 09 17:08:25.604686 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:25.604655 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zfd8z_9efaa331-652f-44ca-9f42-ba4a536de051/kube-rbac-proxy/0.log" Jul 09 17:08:25.628140 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:25.628110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8snr_df15f9e5-7f65-49c2-81e4-9a5ed9bf4c87/dns-node-resolver/0.log" Jul 09 17:08:26.291832 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:26.291803 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxlmb_3d10c6d0-5fa2-469f-87f1-913829110c26/node-ca/0.log" Jul 09 17:08:27.623751 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:27.623719 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-68l6r_172d296a-79ab-4da7-bf3e-f6618d38a54c/payload-processing/0.log" Jul 09 17:08:27.660802 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:27.660766 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6db5c95b-l6zp4_5eb33ab3-d8d8-438c-87c5-d70acafa0db7/router/0.log" Jul 09 17:08:28.470999 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:28.470969 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qnmkm_d43a5652-4e16-4244-87f9-e1acaff4b3fc/serve-healthcheck-canary/0.log" Jul 09 17:08:28.935288 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:28.935251 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-kgtns_c71ed983-0744-4a12-9b54-fd0a633b24d9/insights-operator/0.log" Jul 09 17:08:28.935729 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:28.935676 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-kgtns_c71ed983-0744-4a12-9b54-fd0a633b24d9/insights-operator/1.log" Jul 09 17:08:28.960145 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:28.960121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dffg_5b6b02ae-6d00-43c6-b119-2ee2e78f0887/kube-rbac-proxy/0.log" Jul 09 17:08:28.986046 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:28.986021 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dffg_5b6b02ae-6d00-43c6-b119-2ee2e78f0887/exporter/0.log" Jul 09 17:08:29.009267 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:29.009242 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dffg_5b6b02ae-6d00-43c6-b119-2ee2e78f0887/extractor/0.log" Jul 09 17:08:30.683740 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:30.683714 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7jqtn/perf-node-gather-daemonset-4jc6j" Jul 09 17:08:31.495327 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:31.495294 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bc6967554-zdbsq_13464a1c-7fbf-47f4-84af-ef7cabc3b2da/manager/0.log" Jul 09 17:08:31.818171 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:31.818082 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd696597f-vp54j_18b8ed4e-e86e-466a-b06e-46b52cb3fe92/manager/0.log" Jul 09 17:08:31.840453 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:31.840427 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd696597f-vp54j_18b8ed4e-e86e-466a-b06e-46b52cb3fe92/copy-manifests/0.log" Jul 09 17:08:33.265900 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:33.265867 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d5799cc79-b2v7s_7f40f9f0-d0a3-468c-a43a-7a7aad57dd95/manager/0.log" Jul 09 17:08:33.792869 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:33.792791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-68l6r_172d296a-79ab-4da7-bf3e-f6618d38a54c/payload-processing/0.log" Jul 09 17:08:33.806113 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:33.806082 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6db5c95b-l6zp4_5eb33ab3-d8d8-438c-87c5-d70acafa0db7/router/0.log" Jul 09 17:08:35.250929 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:35.250875 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-68l6r_172d296a-79ab-4da7-bf3e-f6618d38a54c/payload-processing/0.log" Jul 09 17:08:35.273834 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:35.273808 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6db5c95b-l6zp4_5eb33ab3-d8d8-438c-87c5-d70acafa0db7/router/0.log" Jul 09 17:08:38.324296 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:38.324271 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hh2gv_03da5664-7df4-415e-bd93-498c1fb2734e/migrator/0.log" Jul 09 17:08:38.346529 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:38.346500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-hh2gv_03da5664-7df4-415e-bd93-498c1fb2734e/graceful-termination/0.log" Jul 09 17:08:38.691746 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:38.691649 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-gp6cl_edad3ea9-8b2c-4199-be52-06f37aaa936b/kube-storage-version-migrator-operator/1.log" Jul 09 17:08:38.692673 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:38.692631 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-gp6cl_edad3ea9-8b2c-4199-be52-06f37aaa936b/kube-storage-version-migrator-operator/0.log" Jul 09 17:08:39.729994 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.729969 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/kube-multus-additional-cni-plugins/0.log" Jul 09 17:08:39.752766 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.752737 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/egress-router-binary-copy/0.log" Jul 09 17:08:39.776719 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.776692 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/cni-plugins/0.log" Jul 09 17:08:39.801267 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.801237 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/bond-cni-plugin/0.log" Jul 09 17:08:39.825366 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.825338 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/routeoverride-cni/0.log" Jul 09 17:08:39.848796 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.848774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/whereabouts-cni-bincopy/0.log" Jul 09 17:08:39.872001 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:39.871977 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j4j5t_7dbd70a8-4d2e-4f6f-a7ed-4953b91a4a88/whereabouts-cni/0.log" Jul 09 17:08:40.377486 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:40.377453 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qcpwx_d78730c6-cf40-490f-8c55-807f6db8cbf7/kube-multus/0.log" Jul 09 17:08:40.482577 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:40.482543 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dzjz6_a659e30c-c220-4d77-8159-c4d452ea8e95/network-metrics-daemon/0.log" Jul 09 17:08:40.504322 ip-10-0-133-162 kubenswrapper[2577]: I0709 17:08:40.504299 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dzjz6_a659e30c-c220-4d77-8159-c4d452ea8e95/kube-rbac-proxy/0.log"